Combining data from multiple image sensors

ABSTRACT

A method of combining data from multiple sensors is disclosed. The method includes providing a common control signal to multiple image sensors to be synchronized. The method further includes receiving a first data line from a first image sensor of the multiple image sensors, receiving a second data line from a second image sensor of the multiple image sensors, and combining the first data line and the second data line to generate a synchronized data line.

PRIORITY CLAIM

This application claims the benefit of, and incorporates by reference,each of U.S. Provisional Application No. 61/320,940 filed Apr. 5, 2010,U.S. Provisional Application No. 61/324,259 filed Apr. 14, 2010, U.S.Provisional Application No. 61/359,312 filed Jun. 28, 2010, and U.S.Provisional Application No. 61/412,755, filed Nov. 11, 2010.

FIELD

The present disclosure is generally related to combining data frommultiple image sensors.

DESCRIPTION OF RELATED ART

Advances in technology have resulted in smaller and more powerfulcomputing devices. For example, there currently exist a variety ofportable personal computing devices, including wireless computingdevices, such as portable wireless telephones, personal digitalassistants (PDAs), and paging devices that are small, lightweight, andeasily carried by users. More specifically, portable wirelesstelephones, such as cellular telephones and internet protocol (IP)telephones, can communicate voice and data packets over wirelessnetworks. Further, many such wireless telephones include other types ofdevices that are incorporated therein. For example, a wireless telephonecan also include a digital still camera and a digital video camera.Also, such wireless telephones can process executable instructions,including software applications, such as a web browser application, thatcan be used to access the Internet.

In addition, wireless devices may execute three dimensional (3D)applications. In 3D applications, typically at least two image sensorsare used to capture depth information from a scene. Frame data from twoimage sensors is combined and processed to infer distance informationand used to construct a 3D representation. Combining image data fromeach of the sensors typically involves performing frame synchronizationand line synchronization, which may result in synchronization andalignment challenges. In addition, filtering of image data from multiplesensors and interleaving such image data may be further complicated whensource sensors provide data at different frequencies or phases. It wouldbe advantageous to effectively synchronize data from multiple sensorsand efficiently process the data to reduce overall image processingsystem cost and complexity.

SUMMARY

In multiple camera array applications, image data from each of themultiple sensors is to be synchronized at a line level and processed. Animage processing system to combine data from multiple sensors isdisclosed where image data from a first image sensor and a second imagesensor is synchronized and processed. Synchronized data lines aregenerated by synchronizing and combining first data from a first datastream generated by the first image sensor with second data from asecond data stream generated by the second image sensor. The imagesignal processor is configured to process the synchronized data linesreceived from a combiner and to output the processed frame to a display.

In a particular embodiment, a method is disclosed. The method includesproviding a common control signal to multiple image sensors to besynchronized. The method further includes receiving a first data linefrom a first image sensor of the multiple image sensors, receiving asecond data line from a second image sensor of the multiple imagesensors, and combining the first data line and the second data line togenerate a synchronized data line.

In another particular embodiment, an apparatus is disclosed. Theapparatus includes a first input configured to receive a first data linefrom a first image sensor of multiple image sensors to be synchronizedvia a common control signal. The apparatus further includes a secondinput configured to receive a second data line from a second imagesensor of the multiple image sensors, and a combiner coupled to thefirst input and to the second input, wherein the combiner is configuredto combine the first data line and the second data line to generate asynchronized data line.

In another particular embodiment, a method is disclosed. The methodincludes providing a common control signal to multiple image sensors.Each of the multiple image sensors is responsive to the common controlsignal to generate image data. The method further includes receivingsynchronized data output from each of the multiple image sensors,combining the synchronized data output from each of the multiple imagesensors to generate a synchronized data line, and providing thesynchronized data line to an image processor via a single camera inputof the image processor.

In another particular embodiment, an apparatus is disclosed. Theapparatus includes a sensor synchronizer configured to provide a commoncontrol signal to multiple image sensors. Each of the multiple imagesensors is responsive to the common control signal to generate imagedata. The apparatus further includes a combiner configured to combinesynchronized data output received from each of the multiple imagesensors to generate a synchronized data line to be provided to an imageprocessor via a single camera input of the image processor.

In another particular embodiment, a method is disclosed. The methodincludes providing a common control signal to multiple image sensors.Each of the multiple image sensors is responsive to the common controlsignal to generate image data. The method further includes receivingsynchronized data output from each of the multiple image sensors.

In another particular embodiment, a method is disclosed. The methodincludes receiving a common control signal at multiple image sensors.Each of the multiple image sensors is responsive to the common controlsignal to generate image data. The method further includes generatingsynchronized data output from each of the multiple image sensors.

In another particular embodiment, an apparatus is disclosed. Theapparatus includes a sensor synchronizer configured to provide a commoncontrol signal to multiple image sensors to cause the multiple imagesensors to generate image data. The apparatus further includes a sensordata interface configured to receive synchronized data output from eachof the multiple image sensors.

In another particular embodiment, a method is disclosed. The methodincludes receiving lines of image data at an image processor having aninput for a single camera. Each line of the image data includes firstline data from a first image captured by a first camera and second linedata from a second image captured by a second camera. The method furtherincludes generating an output frame having a first section correspondingto line data of the first image and having a second sectioncorresponding to line data of the second image. The first section andthe second section are configured to be used to generate athree-dimensional (3D) image format or a 3D video format.

In another particular embodiment, an apparatus is disclosed. Theapparatus includes an image processor having an input for a singlecamera. The image processor is configured to receive lines of image datavia the input. Each line of the image data includes first line data froma first image captured by a first camera and second line data from asecond image captured by a second camera. The image processor isconfigured to generate an output frame having a first sectioncorresponding to line data of the first image and having a secondsection corresponding to line data of the second image. The firstsection and the second section are configured to be used to generate athree-dimensional (3D) image format or a 3D video format

In a particular embodiment, a method of combining data from multiplesensors into a frame is disclosed. The method includes receiving a firstdata stream from a first image sensor, receiving a second data stream asecond image sensor, and combining data from the first data stream andfrom the second data stream to generate a frame. The method furtherincludes processing the frame at an image signal processor to generate aprocessed frame and outputting the processed frame for display. Each ofthe first image sensor and the second image sensor is directlyresponsive to the image signal processor.

In another particular embodiment, an apparatus is disclosed. Theapparatus includes a first image sensor configured to generate a firstdata stream, a second image sensor configured to generate a second datastream, and a combiner configured to combine first data from the firstdata stream and second data from the second data stream to generate aframe. The apparatus further includes an image signal processorconfigured to process the frame and to output a processed frame to adisplay. Each of the first image sensor and the second image sensor isdirectly responsive to the image signal processor.

In another particular embodiment, a method is disclosed. The methodincludes receiving first image data of an image from a first imagesensor, receiving second image data of an image from a second imagesensor, and synchronizing line by line exposure of the first imagesensor and the second image sensor during image data acquisition. Thefirst image sensor and the second image sensor are independent of eachother. The synchronizing may be line by line and may be frame by frame.

In another particular embodiment, an apparatus is disclosed. Theapparatus includes a memory buffer. The memory buffer includes a sectionto align incoming streams in a deterministic order through the streamingof each frame and a programmable gap section between streams.

In another particular embodiment, a method is disclosed. The methodincludes receiving rows of image data at an image processor having aninput for a single camera. Each row of the image data includes data froma row of a first image captured by a first camera and data from a row ofa second image captured by a second camera. The method also includesgenerating an output having a three dimensional (3D) image format or a3D video format. The output corresponds to the first image and thesecond image.

In another particular embodiment, an apparatus is disclosed. Theapparatus includes an image processor having an input for a singlecamera. The apparatus also includes a combiner configured to send rowsof image data to the image processor. Each row of the image dataincludes first data from a row of a first image captured by a firstcamera and second data from a row of a second image captured by a secondcamera. The image processor is configured to generate an output havingeither a three dimensional (3D) image format or a 3D video format. Theoutput corresponds to the first image and the second image.

One particular advantage provided by at least one of the disclosedembodiments is that a single image signal processor may be used tosynchronize and control image data from multiple image sensors. Anotherparticular advantage is that having gaps between streams offers theflexibility of processing the combined stream in an image signalprocessor as a single frame, and avoids contamination of streams bysubsequent block-based processing (i.e., if the gap is equal with thebiggest block-based processing contamination of streams is avoided).

Other aspects, advantages, and features of the present disclosure willbecome apparent after review of the entire application, including thefollowing sections: Brief Description of the Drawings, DetailedDescription, and the Claims.

BRIE/DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a particular illustrative embodiment of animage processing system to combine data from multiple image sensors;

FIG. 2 is a block diagram of a second illustrative embodiment of animage processing system to combine data from multiple image sensors;

FIG. 3 is a block diagram of a third illustrative embodiment of an imageprocessing system to combine data from multiple image sensors;

FIG. 4 is a block diagram of a particular illustrative embodiment of aselected portion of the image processing system of FIG. 2, where a firstimage sensor and a second image sensor receive common control data;

FIG. 5 is a block diagram of a fourth illustrative embodiment of animage processing system to combine data from multiple image sensors;

FIG. 6 is a diagrammatic representation of a first embodiment of a firstdata stream at an output of a first image sensor and a second datastream at an output of a second image sensor being combined to form asynchronized data line;

FIG. 7 is a diagrammatic representation of a second embodiment of afirst data stream at an output of a first image sensor and a second datastream at an output of a second image sensor being combined to form asynchronized data line;

FIG. 8 is a diagrammatic representation of a first embodiment of phasediagram illustrating a two line phase difference between a first datastream from a first sensor and a second data stream from a secondsensor;

FIG. 9 is a diagrammatic representation of a second embodiment of phasediagram illustrating a one line phase difference between a first datastream from a first sensor and a second data stream from a secondsensor;

FIG. 10 is a diagrammatic representation illustrating pixel data of eachof a plurality of sensors and illustrating synchronized data lines;

FIG. 11 is a timing diagram illustrating frame valid and line validsignal timing for multiple sensors;

FIG. 12 is a diagrammatic representation of a third embodiment of phasediagram illustrating a three line phase difference between a first datastream from a first sensor and a second data stream from a secondsensor;

FIG. 13 is a block diagram of a particular illustrative embodiment of animage processing system to combine data from multiple image sensors toproduce a three dimensional image format;

FIG. 14 is a diagrammatic representation illustrating variousembodiments of mobile devices having image processing systems to combinedata from multiple image sensors;

FIG. 15 is a diagrammatic representation illustrating an example ofimages that are captured by an array of cameras including overlapbetween images;

FIG. 16 is a diagrammatic representation illustrating an example ofimages that are captured by an array of cameras including overlapbetween images where each image may have its own shifting component androtation component;

FIG. 17 is a block diagram illustrating a particular embodiment of anarray of cameras and electrical connections associated with the array ofcameras;

FIG. 18 is a block diagram of a first particular illustrative embodimentof a camera array processing system;

FIG. 19 is a block diagram of a first particular illustrative embodimentof a camera array processing system;

FIG. 20 is a diagrammatic representation illustrating a camera systemthat includes a main lens and multiple cameras arranged in an array;

FIG. 21 is a diagrammatic representation illustrating a multiple cameramodule in an automobile;

FIG. 22 is a flow diagram of a particular illustrative embodiment of amethod of combining data from multiple sensors into a synchronized dataline;

FIG. 23 is a flow diagram of a particular illustrative embodiment of amethod of providing a common control signal to multiple image sensorsand providing a synchronized data line to an image processor via asingle camera input of the image processor;

FIG. 24 is a flow diagram of a particular illustrative embodiment of amethod of providing a common control signal to multiple image sensorsand receiving synchronized data output from each of the multiple imagesensors;

FIG. 25 is a flow diagram of a particular illustrative embodiment of amethod of receiving a common control signal at multiple image sensorsand generating synchronized data output from each of the multiple imagesensors;

FIG. 26 is a flow diagram of a particular illustrative embodiment of amethod of combining data from multiple sensors at an image signalprocessor having an input for a single camera;

FIG. 27 is a flow diagram of a particular illustrative embodiment of amethod of combining data from multiple sensors into a frame;

FIG. 28 is a flow diagram of a particular illustrative embodiment of amethod of synchronizing line by line exposure of a first image sensorand a second image sensor;

FIG. 29 is a flow diagram of a first illustrative embodiment of a methodof combining data from multiple sensors to generate three dimensionalimage data;

FIG. 30 is a flow diagram of a second illustrative embodiment of amethod of combining data from multiple sensors to generate threedimensional image data;

FIG. 31 is a flow diagram of a particular illustrative embodiment of amethod of synchronizing line by line exposure of a first image sensorand a second image sensor;

FIG. 32 is a flow diagram of a particular illustrative embodiment of amethod of combining data from multiple sensors to generate threedimensional image data from a synchronized data line;

FIG. 33 is a block diagram of a particular illustrative embodiment of animage processing system to combine data from multiple image sensors;

FIG. 34 is a block diagram of a first illustrative embodiment of awireless device including an image processing system to combine datafrom multiple image sensors; and

FIG. 35 is a block diagram of a second illustrative embodiment of awireless device including an image processing system to combine datafrom multiple image sensors.

I. DETAILED DESCRIPTION

Referring to FIG. 1, a particular illustrative embodiment of an imageprocessing system to combine data from multiple sensors is depicted andgenerally designated 100. The image processing system 100 includes amultiple camera module 102, a sensor module 104, and a single camerachip module 106. In a particular embodiment, the sensor module 104 mayinclude a plurality of sensors, such as sensors 202 and 204 of FIG. 2and FIG. 3, where each of the plurality of sensors is configured togenerate a data stream that includes data lines of an image. The singlecamera module 106 may include an image processor having a single camerainput, such as image processor 208 of FIG. 2 and FIG. 3. Combiningline-by-line image data of the image data streams of multiple sensorsinto synchronized data lines 108 enables the image processor to performformatting for multi-camera processing even though the image processorhas a single camera input. As a result, the system 100 may beimplemented at a reduced cost as compared to a system that uses aseparate processor for each camera or that uses a processor havingmultiple camera inputs.

Referring to FIG. 2, a particular illustrative embodiment of an imageprocessing system to combine data from multiple sensors is depicted andgenerally designated 200. The image processing system 200 includes afirst sensor 202 and a second sensor 204. The image processing system200 further includes a combiner 206, an image signal processor or videofront end 208, and a sensor synchronizer 230. The image signal processor208 may be coupled to a display device (not shown). The combiner 206includes one or more line buffers 216. The image processing system 200may be integrated in at least one semiconductor die.

The first sensor 202 is configured to generate a first data stream,illustrated as a first image data stream 212. The first image datastream 212 includes a first data line 222. The second sensor 204 isconfigured to generate a second data stream, illustrated as a secondimage data stream 214. The second image data stream 214 includes asecond data line 224. The first and second sensors 202, 204 may besubstantially similar image sensors that are independent of each otherand that receive a common control signal 234 from the sensorsynchronizer 230. The sensor synchronizer 230 is configured to receive acontrol/data signal 232 and to output the common control signal 234 tothe first and second sensors 202, 204, enabling the first and secondsensors 202, 204 to generate closely aligned data streams 212, 214. Forexample, the data streams 212, 214 may have substantially the sametiming characteristics, such as frequency and phase. In a particularembodiment, the control/data signal 232 may be received from the imagesignal processor 208.

The combiner 206 is responsive to the first image data stream 212 andthe second image data stream 214. The combiner 206 is configured tocombine data from the first image data stream 212 and data from thesecond image data stream 214 within the line buffer 216. In a particularembodiment, the line buffer 216 is configured to align first data, suchas the first data line 222 from the first sensor 202, and second data,such as the second data line 224 from the second sensor 204. In aparticular embodiment, the combiner 206 is responsive to data storedwithin the line buffer 216 and provides line data 218 to the imagesignal processor 208. In a particular embodiment, the line data 218 mayinclude a plurality of rows, where each row is a combination ofcorresponding rows from each sensor 202, 204, such as described withrespect to FIG. 6.

The image signal processor 208 is configured to process the line data218 and to generate processed line data 240. In a particular embodiment,the processed line data 240 may be provided as processed frame data.While two sensors have been shown, it should be understood that otherembodiments may include more than two sensors. For example, FIG. 3depicts an embodiment 300 that includes more than two sensors. An Nthsensor 305 is configured to generate an Nth data stream, illustrated asan Nth image data stream 315. The Nth image data stream 315 includes anNth data line 325. The Nth sensor 305 may be substantially similar tothe first and second image sensors 202, 204 and may receive the commoncontrol signal 234 from the sensor synchronizer 230, enabling the first,second, and Nth sensors 202, 204, 305 to generate closely aligned datastreams 212, 214, 315. For example, the data streams 212, 214, 315 mayhave substantially the same timing characteristics, such as frequencyand phase. The combiner 206 is responsive to the first image data stream212, the second image data stream 214, and the Nth image data stream315. The combiner 206 is configured to combine data from the first imagedata stream 212, the second image data stream 214, and the Nth imagedata stream 315 within the line buffer 216.

Because data received from commonly controlled, similar sensors (e.g.,202, 204 of FIG. 2, or 202, 204, 305 of FIG. 3) has substantially thesame frequency and phase, synchronization between data streams receivedat the combiner 206 may occur within a single image line of image data.In a particular embodiment, the line buffer 216 may be dimensioned for aworst case of misalignment (i.e., if the synchronization misalignment isthree lines then the line buffer 212 should be sized to buffer at leastsix lines). As a result, the combined data may be efficiently processedusing a single image signal processor. Thus, overall image system costand complexity may be reduced compared to multiple processor systems(e.g., a processor assigned to each sensor).

Referring to FIG. 4, a particular illustrative embodiment of a selectedportion 400 of the image processing system 200 of FIG. 2 is depicted.The portion 400 of the image processing system includes the first sensor202, the second sensor 204, and the sensor synchronizer 230. In aparticular embodiment, the first sensor 202 and the second sensor 204are identical sensors or nearly identical sensors that receive the samestart-up or reset signals and the same clock input from the sensorsynchronizer 230. For example, the first sensor 202 and the secondsensor 204 may each receive common control data/signals from the sensorsynchronizer 230. In a particular embodiment, the control data/signalsmay include a control clock signal 404, a control data signal 406, acamera clock signal 408, and a camera reset signal 410. The controldata/signals 404-410 may be formed and transmitted via an interfacecompliant with a two wire inter-chip communication protocol, such as anInter-Integrated Circuit (I2C) multi-master serial computer bus.Alternatively, the control data/signals 404-410 may be formed andtransmitted according to an interface compliant with a specification ofa serial interface between a digital camera module and mobile phoneengine, such as a Camera Serial Interface (CSI), an interface between aperipheral device (camera) and a host processor (baseband, applicationengine) such as a Camera Serial Interface 2 (CSI-2), a parallelinterface between a digital camera module and a mobile phone engine suchas a Camera Parallel Interface (CPI), or other control interfaces.

The first sensor 202 may be configured to send first timing data 420 andfirst sensor image data 422 to the combiner 206 as illustrated in thesystem of FIG. 2 or FIG. 5. Similarly, the second sensor 204 may beconfigured to send second timing data 430 and second sensor image data432 to the combiner 206 of FIG. 2 or FIG. 5.

During operation, the first sensor 202 and the second sensor 204 eachoperate in identical or nearly identical conditions from a timingstandpoint. For example, the first and second sensors 202, 204 eachreceive the same control clock signal 404, the same control data signal406, the same camera clock signal 408, and the same camera reset signal410. Because the first and second sensors 202, 204 are identical ornearly identical, they operate substantially similarly under the sametiming conditions. For example, data output from the first sensor 202has substantially the same frequency and phase as data output from thesecond sensor 204. To illustrate, a phase difference between data outputfrom the first sensor 202 and the second sensor 204 may be less than asingle horizontal line of phase difference, enabling a single imagesignal processor to be used to synchronize and control image data fromthe two image sensors 202, 204.

Referring to FIG. 5, a block diagram of a particular illustrativeembodiment of an image processing system to combine data from multiplesensors is depicted and generally designated 500. The system 500includes the first image sensor 202, the second image sensor 204, thecombiner 206, the sensor synchronizer 230, and the image signalprocessor 208. The system 500 further includes a register interface 510and a clock management device 512. In a particular embodiment, theregister interface 510 may be within the sensor synchronizer 230.Alternatively, the register interface 510 may be a standalone module. Ina particular embodiment, the system 500 may further include a thinoutput formatter 506 (shown in shadow) and a transport packer andformatter 508 (shown in shadow).

In a particular embodiment, the combiner 206 is configured to receivethe first timing data 420 and the first sensor image data 422 from thefirst sensor 202. The combiner 206 is also configured to receive thesecond timing data 430 and the second sensor image data 432 from thesecond sensor 204. The combiner 206 is further configured to receive aclock signal 526 from the clock management device 512. The combiner 206uses the first timing data 420, the first sensor image data 422, thesecond timing data 430, and the second sensor image data 432 to generatea synchronized data line which is provided to the image signal processor208. The image signal processor 208 processes the synchronized data lineto create processed data line data. The processed data line data may beprovided to another component, such as to a display device. Thus, imagedata from multiple sensors may be combined, processed and rendered fordisplay at a display device.

In a particular embodiment, the first timing data 420 may be associatedwith a first pixel clock, the first sensor image data 422 may beassociated with a first pixel size, the second timing data 430 may beassociated with a second pixel clock, and the second sensor image data432 may be associated with a second pixel size. When the combiner 406combines the first timing data 420, the first sensor image data 422, thesecond timing data 430, and the second sensor image data 432 to generatethe synchronized data line, a first line of the first image data and acorresponding line of the second image data are combined into a singleimage line. In a particular embodiment, the size of the single imageline may be substantially double that of the first line of the firstimage data or the corresponding line of the second image data (e.g.,double that of the first pixel size or the second pixel size), and therate of pixel clock of the combined single image line may besubstantially double the rate of the first pixel clock or the secondpixel clock (e.g., may have a clock frequency that is double the firstpixel clock frequency or the second pixel clock frequency). Thegenerated synchronized data line is sent to the image signal processor208 via a combiner timing data signal 528 and a combiner image datasignal 530.

In a particular embodiment, the synchronized data line that is generatedby the combiner 206 may be provided to the thin output formatter 506 tocreate formatted data which is provided to the transport packer andformatter 508 prior to being provided to the image signal processor 208.

In a particular embodiment, the thin output formatter 506 receives thecombiner timing data signal 528 and the combiner image data signal 530to create formatted data. The formatted data may include outputformatter timing data signal 536, output formatter image data signal538, output formatter stats data signal 540, output formatter start datasignal 542, and output formatter valid data signal 544. In a particularembodiment, the transport packer and formatter 508 receives theformatted data 536-544 from the thin output formatter 506 and generatesa transport data stream including a transport timing data signal 546 anda transport image data signal 548.

In a particular embodiment, the register interface 510 may be coupled tothe image signal processor 208 and coupled to the clock managementdevice 512. In a particular embodiment, the register interface 510 mayreceive a clock signal 527 from the clock management device 512 and maybe coupled to a register bus 572. The clock management device 512 isconfigured to receive the second timing data signal 430 and to outputthe clock signal 526. In a particular embodiment, the clock signal 526is substantially double the frequency of the second timing data signal430 to enable the combiner 206 to maintain a frame processing rate whilecombining concurrent data from multiple sensors.

Because data output from commonly controlled, similar sensors hassubstantially the same frequency and phase, synchronization between datastreams may occur within a single image line of image data. Thus, thecombined data may be efficiently processed using a single image signalprocessor having access to the single line of image data.

Referring to FIG. 6, a diagrammatic representation of a particularembodiment of a first data stream at an output of a first image sensorand a second data stream at an output of a second image sensor beingcombined to form a synchronized data line is depicted and generallydesignated 600. A first sensor, such as the first sensor 202 of FIG. 2,generates a first data stream 602 that corresponds to first image dataof an image. A second sensor, such as the second sensor 204 of FIG. 2,generates a second data stream 604 that corresponds to second image dataof the image. Data from the first data stream 602 and data from thesecond data stream 604 are combined to form a data out data stream 606.

In a particular embodiment, the first data stream 602 includes dataassociated with a first line of the first image data of the image andthe second data stream 604 includes data associated with a correspondingline of the second image data of the image. The first data stream 602includes line data 610 having a first line index value, line data 612having a second line index value, line data 614 having a third lineindex value, and line data 616 having a fourth line index value. Thesecond data stream 604 includes corresponding line data to that of thefirst data stream, including corresponding line data 620 having thefirst line index value, corresponding line data 622 having the secondline index value, corresponding line data 624 having the third lineindex value, and corresponding line data 626 having the fourth lineindex value.

The data out data stream 606 includes a combination of the first line ofthe first image data of the image and the corresponding line of thesecond image data of the image. As illustrated, the first data stream602 and the second data stream 604 are interleaved to form the data outdata stream 606. For example, the data out data stream 606 includescombined line data 630 having the first line index value, combined linedata 632 having the second line index value, and combined line data 634having the third line index value. The combined line data 630 includesthe line data 610 and the corresponding line data 620. The combined linedata 632 includes the line data 612 and the corresponding line data 622.The combined line data 634 includes the line data 614 and thecorresponding line data 624. Each combined line 630-634 may be generatedby combining corresponding lines within a line buffer, such as the linebuffer 216 of FIG. 2.

In a particular embodiment, the data from the first data stream 602 iscombined with the data from the second data stream 604 to generate aplurality of synchronized data lines that form a frame 660. The frame660 may include a plurality of rows 642, where each row corresponds to aline index value and stores a synchronized data line that includes aline of the first image data having the line index value and acorresponding line of the second image data having the line index value.For example, a first row of the frame 660 may include the combined linedata 630, a second row of the frame 660 may include the combined linedata 632, a third row of the frame 660 may include the combined linedata 634, etc. Each synchronized image data line forms part of the frame660 such that the data in the frame 660 is aligned.

The frame 660 is depicted with an order of the rows 642 matching a readorder of the image data from the image sensors (i.e. combined data fromthe top line of the image sensors (line index 1) is in a top line of theframe 660 and combined data from a next line of the image sensors (lineindex 2) is in a next line of the frame 660. Alternatively, in otherembodiments the rows of the frame 660 may not match a read order of theimage data and may instead correspond to any other order of the imagedata. For example, a top row of the frame 660 may to line index 2 whilea next row of the frame 660 may correspond to line index 1. The frame660 may be programmable such that each of the rows 642 can be programmedto correspond to any of the line index values of the image data.

In a particular embodiment, the first line 610 of the first image datais associated with a first pixel size (e.g., a number of pixels perline) and a first pixel clock, and the corresponding line 620 of thesecond image data is associated with a second pixel size and a secondpixel clock. The first sensor and the second sensor generating the datastreams 602, 604 are configured to receive a common clock signal and acommon reset signal. When the first line 610 of the first image data andthe corresponding line 620 of the second image data are combined into asingle image line, the size of the single image line is substantiallydouble that of the first line 610 of the first image data or thecorresponding line 620 of the second image data, and the pixel clocksignal of the combined single image line (e.g., a third pixel clocksignal) has a clock rate that is substantially double that of the firstpixel clock signal or the second pixel clock signal. For example, thecombined line data 630 may have an image size that is substantiallydouble that of the line data 610 or double that of the correspondingline data 620. Similarly, the pixel clock frequency of the combined linedata 630 may have a frequency that is substantially double that of thefirst pixel clock signal associated with the line data 610 or doublethat of the second pixel clock signal associated with the correspondingline data 620, such that the pixel clock frequency of the combined linedata 630 may be associated with the third pixel clock signal havingsubstantially double the frequency of that of the first pixel clock orthe second pixel clock.

Alternatively, in an implementation where line data three image sensorshaving equal line sizes are combined, a synchronized line size may besubstantially three times the sensor line size and a pixel clock ratemay be substantially three times a pixel clock rate of the individualsensors. In a general case of an arbitrary number of sensors that mayhave unequal sizes, a synchronized line size can be set as greater thanor equal to a sum of the line sizes that are combined, and a pixel clockrate can be set so that the output line bandwidth is equal to or greaterthan the sum of the input bandwidth.

The frame 660 may be processed at an image signal processor, such as theimage signal processor 208 of FIG. 2, to generate a processed frame 650.The processed frame 650 includes a first section 652 including firstimage data from a first image sensor, such as the sensor 202 of FIG. 2,a second section 654 including second image data from a second imagesensor, such as the sensor 204 of FIG. 2, and a gap section 656. The gapsection 656 may include non-image data disposed between the firstsection 652 and the second section 654.

In a particular embodiment, the first section 652 includes a line of thefirst image data and the second section 654 includes a correspondingline of the second image data. In a particular embodiment, the gapsection 656 may be used for edge filtering and may include a black gapthat is approximately five pixels in width. As a further example, thegap section 656 may be added between lines and have a size equal to thesize of an interpolation kernel or a size of a largest two-dimensionalfilter applied to the frame 650 by the image signal processor.

In a particular illustrative embodiment, statistics for automaticexposure, automatic focus, and automatic white balance may be collectedfrom either the first section 652 or the second section 654, either ofwhich may be a full image from one of the respective sensors. Therefore,the statistics for automatic exposure, automatic focus, and automaticwhite balance may be collected from half of the final image (e.g., thefirst section 652) and may be applied to both sensors since both sensorsare receiving substantially identical timing information. As such, dataoutput from multiple sensors has substantially the same frequency andphase such that synchronization may occur within one image line of imagedata of the image.

The frame 650 may be stored in a memory that is integrated in at leastone semiconductor die. The frame 650 may be stored in memory that isincorporated into a consumer electronic device, such as a set top box, amusic player, a video player, an entertainment unit, a navigationdevice, a communications device, a personal digital assistant (PDA), afixed location data unit, and a computer. The electronic devices mayutilize image processing methods, including 3D applications that processimage data from multiple sensors.

Referring to FIG. 7, a diagrammatic representation of a secondembodiment of a first data stream at an output of a first image sensorand a second data stream at an output of a second image sensor beingcombined to form a synchronized data line is depicted and generallydesignated as 700. A first sensor, such as the first image sensor 202 ofFIG. 2, generates a first data stream 702 that corresponds to firstimage data of an image. A second sensor, such as the second image sensor204 of FIG. 2, generates a second data stream 704 that corresponds tosecond image data of the image. Data from the first data stream 702 anddata from the second data stream 704 are combined to form a data outdata stream 706.

In a particular embodiment, the first data stream 702 includes dataassociated with a first line of the first image data of the image andthe second data stream 704 includes data associated with a correspondingline of the second image data of the image. The first data stream 702includes line data 710 having a first line index value, line data 712having a second line index value, line data 714 having a third lineindex value, and line data 716 having a fourth line index value. Thesecond data stream 704 includes corresponding line data to that of thefirst data stream, including corresponding line data 720 having thefirst line index value, corresponding line data 722 having the secondline index value, corresponding line data 724 having the third lineindex value, and corresponding line data 726 having the fourth lineindex value.

The data out data stream 706 includes a combination of the first line ofthe first image data of the image and the corresponding line of thesecond image data of the image. As illustrated, the first data stream702 and the second data stream 704 are interleaved with a gap section708 to form the data out data stream 706. For example, the illustratedportion of the data out data stream 706 includes combined line data 730having the first line index value, combined line data 732 having thesecond line index value, and combined line data 734 having the thirdline index value. The combined line data 730 includes the line data 710separated from the corresponding line data 720 by the gap section 708.The combined line data 732 includes the line data 712 separated from thecorresponding line data 722 by the gap section 708. The combined linedata 734 includes the line data 714 separated from the correspondingline data 724 by the gap section 708. Each combined line 730-734 may begenerated by combining corresponding lines with the gap section 708between the corresponding lines within a line buffer, such as the linebuffer 216 of FIG. 2.

In a particular embodiment, the data from the first data stream 702 iscombined with the data from the second data stream 704 to generate aplurality of synchronized data lines that form a frame 740. The frame740 may include a plurality of rows 742, where each row corresponds to aline index value and stores a line of the first image data having theline index value and stores a corresponding line of the second imagedata having the line index value. For example, a first row of the frame740 may include the combined line data 730, a second row of the frame740 may include the combined line data 732, a third row of the frame 740may include the combined line data 734, etc. such that the data in theframe 740 is aligned.

In a particular embodiment, the first line 710 of the first image datais associated with a first pixel size (e.g., a number of pixels perline) and a first pixel clock, and the corresponding line 720 of thesecond image data is associated with a second pixel size and a secondpixel clock. The first sensor and the second sensor generating the datastreams 702, 704 are configured to receive a common clock signal and acommon reset signal. When the first line 710 of the first image data andthe corresponding line 720 of the second image data are combined into asingle image line, the size of the single image line is approximatelydouble that of the first line 710 of the first image data or thecorresponding line 720 of the second image data. Further, the pixelclock signal of the combined single image line (e.g., a third pixelclock signal) has a clock rate that is approximately double that of thefirst pixel clock signal or the second pixel clock signal. For example,the combined line data 730 may have an image size that is approximatelydouble that of the line data 710 or double that of the correspondingline data 720. Similarly, the pixel clock frequency of the combined linedata 730 may have a frequency that is approximately double that of thefirst pixel clock signal associated with the line data 710 or doublethat of the second pixel clock signal associated with the correspondingline data 720, such that the pixel clock frequency of the combined linedata 730 may be associated with the third pixel clock signal havingapproximately double the frequency of that of the first pixel clock orthe second pixel clock.

Each gap section 708 may include non-image data. In a particularembodiment, the non-image data area in the frame 740 formed by the gapsections 708 may be used for edge filtering. The gap section 708 mayinclude a black gap that is approximately five pixels in width. In otherembodiments, each gap section 708 has a size equal to the size of aninterpolation kernel or a size of a largest two-dimensional filterapplied to the frame 740 by an image processor, such as the imageprocessor 208 of FIG. 2. The frame 740 may be processed by the imageprocessor to generate a 3D image.

Referring to FIG. 8, a diagrammatic representation of a first embodimentof a phase diagram illustrating a two line phase difference between afirst data stream from a first sensor and a second data stream from asecond sensor is depicted and generally designated 800. A first sensor,such as the first sensor 202 of FIG. 2, generates a first data streamthat includes first sensor first line data 802, first sensor second linedata 804, and first sensor third line data 806. A second sensor, such asthe second sensor 204 of FIG. 2, generates a second data stream thatincludes second sensor first line data 812, second sensor second linedata 814, and second sensor third line data 816. Data from the firstdata stream and data from the second data stream are combined to form acombined line 820. In the particular embodiment illustrated in FIG. 8,first, second and third line data is illustrated. Alternatively, anynumber of line data may be generated (e.g., 720 lines as illustrated inFIGS. 6 and 7).

In the particular embodiment of FIG. 8, there is a two line phasedifference between data of the first data stream and data of the seconddata stream. For example, the first sensor first line data 802 may bereceived by a combiner such as combiner 216 of FIG. 2 during a firstline phase, the first sensor second line data 804 may be received duringa second line phase, and the first sensor third line data 806 and thesecond sensor first line data 812 may be received during a third linephase. As such, there is a two line phase difference between the firstsensor first line data 802 and the second sensor first line data 812.

The combined line 820 includes a combination of the first line of thefirst image data of the image and the corresponding line of the secondimage data of the image. As illustrated, the first data stream and thesecond data stream are interleaved to form the combined line 820. Forexample, the combined line 820 includes combined line data 822 havingthe first sensor first line data 802 and the second sensor first linedata 812, combined line data 824 having the first sensor second linedata 804 and the second sensor second line data 814, and combined linedata 826 having the first sensor third line data 806 and the secondsensor third line data 816. Each combined line 822-826 may be generatedby combining corresponding lines within a line buffer, such as the linebuffer 216 of FIG. 2. The line buffer may be configured to buffer atleast a portion of a next data line (e.g. the first sensor second linedata 804) that is received from the first image sensor before asynchronized data line (e.g. the combined line data 822) is generated.As such, data output from multiple sensors having a two line phasedifference may be combined such that synchronization may occur withinone image line of image data of the image.

Referring to FIG. 9, a diagrammatic representation of a first embodimentof a phase diagram illustrating a one line phase difference between afirst data stream from a first sensor and a second data stream from asecond sensor is depicted and generally designated 900. A first sensor,such as the first sensor 202 of FIG. 2, generates a first data streamthat includes first sensor first line data 902, first sensor second linedata 904, first sensor third line data 906, and first sensor fourth linedata 908. A second sensor, such as the second sensor 204 of FIG. 2,generates a second data stream that includes second sensor first linedata 912, second sensor second line data 914, second sensor third linedata 916, and second sensor fourth line data 918. Data from the firstdata stream and data from the second data stream are combined to form acombined line 920. In the particular embodiment illustrated in FIG. 9,first, second, third, and fourth line data is illustrated.Alternatively, any number of line data may be generated (e.g., 720 linesas illustrated in FIGS. 6 and 7).

In the particular embodiment of FIG. 9, there is a one line phasedifference between data of the first data stream and data of the seconddata stream. For example, the first sensor first line data 902 may bereceived by a combiner such as combiner 216 of FIG. 2 during a firstline phase, and the first sensor second line data 904 and the secondsensor first line data 912 may be received during a second line phase.As such, there is a one line phase difference between the first sensorfirst line data 902 and the second sensor first line data 912.

The combined line 920 includes a combination of the first line of thefirst image data of the image and the corresponding line of the secondimage data of the image. As illustrated, the first data stream and thesecond data stream are interleaved to form the combined line 920. Forexample, the combined line 920 includes combined line data 922 havingthe first sensor first line data 902 and the second sensor first linedata 912, combined line data 924 having the first sensor second linedata 904 and the second sensor second line data 914, combined line data926 having the first sensor third line data 906 and the second sensorthird line data 916, and combined line data 928 having the first sensorfourth line data 908 and the second sensor fourth line data 918. Eachcombined line 922-926 may be generated by combining corresponding lineswithin a line buffer, such as the line buffer 216 of FIG. 2. As such,data output from multiple sensors having a one line phase difference maybe combined such that synchronization may occur within one image line ofimage data of the image.

Referring to FIG. 10, a diagrammatic representation illustratingred-green-blue (RGB) data for each of a plurality of sensors andillustrating synchronized data lines is depicted and generallydesignated 1000. In the embodiment illustrated in FIG. 10, a firstsensor, such as the first sensor 202 of FIG. 3, generates a first datastream that includes first sensor first line data 1002, first sensorsecond line data 1004, and first sensor third line data 1006. A secondsensor, such as the second sensor 204 of FIG. 3, generates a second datastream that includes second sensor first line data 1012, second sensorsecond line data 1014, and second sensor third line data 1016. A thirdsensor, such as the Nth sensor 305 of FIG. 3, generates a third datastream that includes third sensor first line data 1022, third sensorsecond line data 1024, and third sensor third line data 1026. Data fromthe first data stream, data from the second data stream, and data fromthe third data stream are combined to form a combined line 1020.

As an illustrative example, each of the first line data 1002, 1012, 1022includes alternating red and green pixel values, each of the second linedata 1004, 1014, 1024 includes alternating green and blue pixel values,and each of the third line data 1006, 1016, 1026 includes alternatingred and green pixel values according to a Bayer filter pattern.

As illustrated, the first data stream, the second data stream, and thethird data stream are interleaved to form a combined line data stream1020. For example, the combined line data stream 1020 includes combinedline data 1040 having the first sensor first line data 1002, the secondsensor first line data 1012, and the third sensor first line data 1002,combined line data 1050 having the first sensor second line data 1004,the second sensor second line data 1014, and the third sensor secondline data 1024, and combined line data 1060 having the first sensorthird line data 1006, the second sensor third line data 1016, and thethird sensor third line data 1026. Each combined line 1040-1060 may begenerated by combining corresponding lines within a line buffer, such asthe line buffer 216 of FIG. 3. Each combined line 1040-1060 may includeraw pixel (e.g., R, G, and B) values read from its respective sensorline data to be demosaiced at an image signal processor, such as theimage signal processor 208 of FIG. 3. As such, data output from multiplesensors may be combined such that synchronization may occur within oneimage line of image data of the image.

Although FIG. 10 illustrates raw pixel values as including RGB dataaccording to a Bayer filter pattern, in other embodiments the line datafrom the sensors may not include Bayer RGB pixel values. For example,the sensors may instead provide: luma, blue-difference chroma,red-difference chroma (YCbCr) values; cyan, yellow, green, and magenta(CYGM) values; red, green, blue, and emerald (RGBE) values; red, green,blue, and white (RGBW) values; or any other type of values, asillustrative, non-limiting examples. Alternatively, or in addition, oneor more of the sensors may implement panchromatic cells, microlensesover groups of photoreceptors, vertical color filters, or any othersensor technology capable of line-by-line readout of raw image data.

Referring to FIG. 11, a particular embodiment of a timing diagramillustrating signal timing associated with an apparatus having multiplesensors is depicted and generally designated 1100. The signals include aframe valid (FV) signal 1102 and a line valid (LV) signal 1104 of afirst sensor, an FV 1106 and an LV 1108 of a second sensor, and an FV1110 and an LV 1112 of a third sensor. In a particular embodiment, thefirst sensor, the second sensor, and a third sensor may be the first,second, and third sensors 202, 204, 305 of FIG. 3. A combined framevalid signal 1114 is also illustrated in conjunction with a combinedline valid/data signal 1116 and a line load (LL) signal 1118.

In a particular embodiment, the signals 1114-1118 correspond tosignaling related to one or more synchronized data lines of a frame,such as the frame 660 of FIG. 6 generated by a combiner, and the signals1102-1112 correspond to signaling received at the combiner. Asillustrated, a first line data 1120, a second line data 1122, and athird line data 1124 are received from the first sensor, a first linedata 1130, a second line data 1132, and a third line data 1134 arereceived from the second sensor, and a first line data 1140, a secondline data 1142, and a third line data 1144 are received from the thirdsensor.

The first line data 1130 is received from the second sensor prior to thefirst line data 1120 and the first line data 1140. A phase differencebetween receipt of the first line data 1130 and the second line data1120 is illustrated as a first phase difference 1180. The first linedata 1120 of the first sensor is received prior to the first line data1140 of the third sensor, illustrated as a second phase difference 1182.The line data from each of the sensors may follow a rising edge of acorresponding frame valid signal, indicating that data received via animage data line is valid line data from each particular sensor. Asillustrated, the combined frame valid line 1114 remains low, indicatingnon-valid data, until after each of the first line data 1120, 1130, and1140 have been received, such as at the combiner 206 of FIG. 2. Aftereach of the first lines 1120, 1130, and 1140 has been received, thecombined frame valid signal 1114 rises to indicate valid data on theline valid signal 1116. A first synchronized data line 1150 is generatedin conjunction with a valid signal being asserted on the LL line 1118.After generation of the first synchronized line data 1150, the combinedframe valid signal 1114 remains in a valid state while the LL signal1118 returns to a non-valid state, after which the LL signal 1118returns to a valid state upon generation of a second synchronized dataline 1160, which is followed by generation of a third synchronized dataline 1170.

Referring to FIG. 12, a diagrammatic representation of a firstembodiment of a phase diagram illustrating a three line phase differencebetween a first data stream from a first sensor, a second data streamfrom a second sensor, and a third data stream from a third sensor isdepicted and generally designated 1200. A first sensor, such as thefirst sensor 202 of FIG. 3, generates a first data stream that includesfirst sensor first line data 1202, first sensor second line data 1204,first sensor third line data 1206, and first sensor fourth line data1208. A second sensor, such as the second sensor 204 of FIG. 3,generates a second data stream that includes second sensor first linedata 1212, second sensor second line data 1214, second sensor third linedata 1216, and second sensor fourth line data 1218. A third sensor, suchas the Nth sensor 305 of FIG. 3, generates a third data stream thatincludes third sensor first line data 1222, third sensor second linedata 1224, third sensor third line data 1226, and third sensor fourthline data 1228. Data from the first data stream, data from the seconddata stream, and data from the third data stream are combined to form acombined line 1220. In the particular embodiment illustrated in FIG. 12,first, second, third, and fourth line data is illustrated.Alternatively, any number of line data may be generated (e.g., 720 linesas illustrated in FIGS. 6 and 7).

In a particular embodiment, there is a three line phase differencebetween data of the first data stream, data of the second data stream,and data from the third data stream. For example, the third sensor firstline data 1222 may be received by a combiner such as combiner 216 ofFIG. 2 during a first line phase, the first sensor first line data 1202and the third sensor second line data 1224 may be received during asecond line phase, the first sensor second line data 1204 and the thirdsensor third line data 1226 may be received during a third line phase,and the first sensor third line data 1206, the second sensor first linedata 1212, and the third sensor fourth line data 1228 may be receivedduring a fourth line phase. As such, there is a three line phasedifference between the first sensor first line data 1202, the secondsensor first line data 1212, and the third sensor first line data 1222.

The combined line 1220 includes a combination of the first line of thefirst image data of the image and the corresponding line of the secondimage data and the third image data of the image. As illustrated, thefirst data stream, the second data stream, and the third data stream areinterleaved to form the combined line 1220. For example, the combinedline 1220 includes combined line data 1232 having the first sensor firstline data 1202, the second sensor first line data 1212, and the thirdsensor first line data 1222, combined line data 1234 having the firstsensor second line data 1204, the second sensor second line data 1214,and the third sensor second line data 1224, and combined line data 1236having the first sensor third line data 1206, the second sensor thirdline data 1216, and the third sensor third line data 1226. Each combinedline 1232-1236 may be generated by combining corresponding lines withina line buffer, such as the line buffer 216 of FIG. 3. As such, dataoutput from multiple sensors having a three line phase difference may becombined such that synchronization may occur within one image line ofimage data of the image.

Referring to FIG. 13, a particular illustrative embodiment of an imageprocessing system to combine data from multiple sensors to generate 3Ddata is depicted and generally designated 1300. The image processingsystem 1300 includes a first image sensor 1302 and a second image sensor1304. The image processing system 1300 further includes a combiner 1306and an image processor 1308. The image processor 1308 is coupled to adisplay device 1310. The combiner 1306 includes at least one line buffer1312. The image processing system 1300 may be integrated in at least onesemiconductor die.

The first image sensor 1302 is configured to generate a first datastream, illustrated as a first image data stream 1314, and the secondimage sensor 1304 is configured to generate a second data stream,illustrated as a second image data stream 1316. In a particularembodiment, the first image data stream 1314 may be asynchronous to thesecond image data stream 1316. The first and second image sensors 1302,1304 may be substantially similar image sensors that are independent ofeach other and that may receive a common control signal from a processor(e.g., the combiner 1306 or the image processor 1308) to generateclosely aligned image data streams 1314, 1316. For example, the imagedata streams 1314, 1316 may have substantially the same timingcharacteristics, such as frequency and phase. Because the common controlsignal may be generated by the processor, such as via the signalsynchronizer 230 of FIG. 2, each of the image sensors 1302, 1304 may bedirectly responsive to, and controlled by, the single processor. Whiletwo image sensors 1302, 1304 have been shown, it should be understoodthat more than two image sensors may be used with the image processingsystem 1300.

The combiner 1306 is responsive to the first image data stream 1314 andthe second image data stream 1316. The combiner 1306 is configured tocombine data from the first image data stream 1314 and data from thesecond image data stream 1316 within the line buffer 1312. In aparticular embodiment, the line buffer 1312 is configured to align firstdata from the first image sensor 1302 and second data from the secondimage sensor 1304. In a particular embodiment, the combiner 1306 isresponsive to data stored within the line buffer 1312 and provides framedata 1318 to the image processor 1308. In a particular embodiment, theframe data 1318 may include a plurality of rows of image data, whereeach row is a combination of corresponding rows from each sensor 1302,1304, such as described with respect to FIG. 3.

The image processor 1308 is configured to process the frame data 1318and to output processed frame data 1320 to the display device 1310. Theprocessed frame data 1320 may have a 3D image format or a 3D videoformat.

The display device 1310 renders and displays image data in response toreceiving the processed frame data 1320. Thus, image data from multipleimage sensors may be combined, processed and then rendered for displayat the display device 1310. In a particular embodiment, the displaydevice 1310 may be decoupled from the image processor 1308 to not bedirectly responsive to the image processor 1308. For example, thedisplay device 1310 may be a separate device from the image processor1308.

Because data received from the commonly controlled, similar imagesensors 1302, 1304 may have substantially the same frequency and phase,synchronization between the data streams 1314, 1316 may occur within asingle image line of image data. In a particular embodiment, the linebuffer 1312 may be dimensioned for a worst case of misalignment (e.g.,if the synchronization misalignment is three lines, then the line buffer1312 should be sized to store at least six lines). As a result, thecombined data may be efficiently processed using a single imageprocessor 1308. Thus, overall image system cost and complexity may bereduced compared to multiple processor systems (e.g., a processorassigned to each sensor).

Embodiments may be configured to provide 3D/stereoscopic images and/orvideo data. For example, in one such embodiment, the first image sensor1302 and the second image sensor 1304 may be positioned side by side soas to provide left/right (stereoscopic) images. The signal provided bythe combiner 1306 is received and may be processed by the imageprocessor 1308 to produce 3D images. A user command may allow the imageprocessor 1308 to receive and process data from only a single sensor(i.e., the first image sensor 1302 or the second image sensor 1304) toproduce two dimensional (2D) images in lieu of producing 3D images.

An image processor having an input for a single camera, such as imageprocessor 1308, is able to process data that can be used for 3Dprocessing by using combined data from two cameras provided by thecombiner 1306. The image processor 1308 may receive image data from thecombiner 1306 or from a memory that stores image data from the combiner1306. In one such embodiment, the image processor 1308 processesreceived image data as 2D image/video data so that subsequent processingby the image processor 1308 provides a 3D stereoscopic image/videostream based on the processed data from the image processor 1308.Alternatively, the image processor 1308 may be configured to directlyprovide a 3D stereoscopic image/video stream based on received imagedata. In one embodiment, a 3D capture system comprises the combiner 1306implemented as a first integrated circuit and the image processor 1308implemented as a second integrated circuit. The first and secondintegrated circuits may be connected, for example, by one or more of aserial, parallel, or I2C bus.

Combining the image data stream 1314, 1316 to generate the frame data1318 enables the image processor 1308 to perform formatting formulti-camera 3D processing even though the image processor 1308 has asingle camera input. As a result, the system 1300 may be implemented ata reduced cost as compared to a system that use a separate processor foreach camera or that uses a processor having multiple camera inputs.

Referring to FIG. 14, a diagrammatic representation illustrating variousembodiments of mobile devices having image processing systems to combinedata from multiple image sensors is depicted and generally designated1400. In the particular embodiment illustrated at 1402, the mobiledevice includes an image processing system having an array of twoadjacent cameras. In the particular embodiment illustrated at 1404, themobile device includes an image processing system having an array ofthree cameras arranged in an in-line configuration. Alternatively, anynumber of cameras may be arranged in an in-line configuration. In theparticular embodiment illustrated at 1406, the mobile device includes animage processing system having a three-by-three array of cameras. In theparticular embodiment illustrated at 1408, the mobile device includes animage processing system having a five-by-five array of cameras.Alternatively, any number of cameras may be arranged in atwo-dimensional array.

Referring to FIG. 15, an example of an array of images that are capturedby multiple cameras or image sensors is illustrated and generallydesignated 1500. As illustrated in FIG. 15, an image captured by onecamera may overlap with images captured by other neighboring cameras.Image overlap may be useful in combining the images captured by each ofthe cameras into a single image. In the particular embodimentillustrated in FIG. 15, the array 1500 corresponds to a four-by-fourarray of cameras or image sensors. Alternatively, any number of camerasmay be arranged in a two-dimensional array (e.g., a five-by-five arrayas illustrated in FIGS. 18 and 19).

Each camera captures a single camera image of the array 1500. In theembodiment illustrated in FIG. 15, the array 1500 is a four-by-fourarray. As such, the array 1500 includes a first row 1502, a second row1504, a third row 1506, and a fourth row 1508. Further, the array 1500includes a first column 1510, a second column 1512, a third column 1514,and a fourth column 1516. As an illustrative example, a single cameraimage 1518 may be captured by a camera corresponding to the first row1502 and the fourth column 1516. The single camera image 1518 mayoverlap with camera images captured by other cameras of neighboringcells. In the case of the camera corresponding to the first row 1502 andthe fourth column 1516, a camera of a neighboring cell may include acamera corresponding to the first row 1502 and the third column 1514, acamera corresponding to the second row 1504 and the third column 1514,or a camera corresponding to the second row 1504 and the fourth column1516. For example, a single camera image overlap 1520 may be associatedwith the single camera image 1508 captured by a camera corresponding tothe first row 1502 and the fourth column 1516. FIG. 15 illustrates atheoretical case of absolute alignment of each camera of each of therows 1502-1508 and each camera of each of the columns 1510-1516. Assuch, the image overlap with an image captured by a camera correspondingto a neighboring row (e.g., the second row 1504) may be the same as theimage overlap with an image captured by a camera corresponding to aneighboring column (e.g., the third column 1514).

An individual image may be captured with a particular horizontalresolution (“H-res”). To illustrate, a horizontal resolution 1522 may beassociated with the single camera image 1518 captured by the cameracorresponding to the first row 1502 and the fourth column 1516. FIG. 15illustrates a theoretical case of image overlap where each camera hasthe same horizontal resolution 1522. As such, an overall horizontalresolution 1524 (i.e., a number of pixels in each of the rows 1502-1508)may be calculated based on a combination of the horizontal resolution1522 of an individual camera and a number of cells in a row (“n”). Inthe case of the four-by-four array 1500 of FIG. 15, there are four cellsin each row (i.e., a cell associated with each of the columns1510-1516). The overall horizontal resolution 1524 may account for imageoverlap. For example, the overall horizontal resolution 1524 may becalculated as H_res*n−overlap*(n−2), where overlap indicates a number ofoverlapping pixels of adjacent images. A similar calculation may beperformed to determine an overall vertical resolution.

FIG. 15 illustrates a theoretical case of absolute alignment of eachcamera with the same image overlap. The image overlap between imagescaptured by the individual cameras may allow each of the individualimages to be “stitched together” into a single image.

Referring to FIG. 16, an example of an array of images that are capturedby multiple cameras or image sensors is illustrated and generallydesignated 1600. As illustrated in FIG. 16, an image captured by onecamera may overlap with images captured by other neighboring cameras.Image overlap may be useful in combining the images captured by each ofthe cameras into a single image. In the particular embodimentillustrated in FIG. 16, the array 1600 corresponds to a four-by-fourarray of cameras or image sensors. Alternatively, any number of camerasmay be arranged in a two-dimensional array. FIG. 16 illustrates that,due to mechanical constraints, it may not be feasible to achieveabsolute alignment of cameras in a row or column (as illustrated in FIG.15).

Each image of the array 1600 may have its own rotation 1602, shift 1604,and tilt (not shown). One or more positioning tolerances 1606 may beassociated with each image. The positioning tolerances 1606 may includea rotation tolerance, a shift tolerance, a tilt tolerance, or acombination thereof. Image overlap may be useful in combining the imagescaptured by each of the cameras into a single image. FIG. 16 illustratesthat the image overlap that is used in combining the images may accountfor the rotation 1602, shift 1604, and tilt of each image that resultsfrom mechanical constraints associating with building a device. Once thedevice is built, the image overlap may be known and stable. As such, theimage overlap may be quantified and may be corrected in a later stage.

Referring to FIG. 17, a particular embodiment of an array of cameras andelectrical connections associated with the array is illustrated andgenerally designated 1700. FIG. 17 illustrates that each camera of thearray 1700 has a first type of interface (i.e., a data interface 1702)and a second type of interface (i.e., a control interface 1704). In theembodiment illustrated in FIG. 17, the array 1700 includes athree-by-three array of cameras. Alternatively, the array 1700 mayinclude cameras arranged in a four-by-four array (e.g., the arrays 1500,1600 of FIGS. 15 and 16) or any other number of cameras arranged in atwo-dimensional array.

The data interface 1702 may include a serial data bus (e.g., a MobileIndustry Processor Interface or a Standard Mobile Imaging Architectureinterface). The data interface 1702 in FIG. 17 is associated with afirst row 1706 of the array 1700, a second row 1708 of the array 1700,and a third row 1710 of the array 1700. Lines associated with the datainterface 1702 may be used to collect data from cameras in each of therows 1706-1710 to be processed in parallel. As an illustrative example,for a resolution up to five megapixels (mpix) and ten frames per second,four wires may be needed (e.g., differential data and clock).

In the illustrative three-by-three array 1700 of FIG. 17, each of therows 1706-1710 includes a camera in a first column 1712 of the array1700, a camera in a second column 1714 of the array 1700, and a camerain a third column 1716 of the array 1700. Thus, the data interface 1702may be used to collect data from nine cameras to be processed inparallel.

The control interface 1704 may include lines that are used tosynchronize all cameras in the array 1700. For example, controlinterface lines may be associated with clock, reset, and I2Ccommunication. In the illustrative three-by-three array 1700 of FIG. 17,the control interface 1704 may be used to synchronize the nine camerasof the array 1700.

Referring to FIG. 18, a particular embodiment of a camera arrayprocessing system is illustrated and generally designated 1800. Allcameras in an array 1802 may be responsive to common control signals,aligned, and processed prior to the resulting image data being combinedinto a final image. In the embodiment illustrated in FIG. 18, the array1802 includes a five-by-five array of cameras. Alternatively, the array1802 may include any other number of cameras arranged in atwo-dimensional array (e.g., the three-by-three array 1700 of FIG. 17).

All cameras in the array 1802 may be synchronized using a method ofconcurrent image sensor support using a single ISP pipeline. Further,each row of cameras may be aligned using an alignment method. That is,one row of images may be collected, aligned in the same order and sentfor processing as a single line with a size n*line, where n is thenumber of cameras in a row and line is the horizontal size (i.e.,“H_res” as described in FIG. 15) of one camera. The five-by-five array1802 illustrated in FIG. 18 includes a first row 1804, a second row1806, a third row 1808, a fourth row 1810, and a fifth row 1812.Further, the five-by-five array 1802 includes a first column 1814, asecond column 1816, a third column 1818, a fourth column 1820, and afifth column 1822. Each of the rows 1804-1812 may be processed inparallel for color at a VFE component, and each of the rows 1804-1812may be processed at a graphics processor or graphics processing unit(GPU) component to align and rectify each individual image in a row.After rectification and alignment, GPU processing may be performed tocombine all of the rows 1804-1812 together, resulting in a final image.

The first row 1804 may be associated with a first alignment block 1824,the second row 1806 may be associated with a second alignment block1826, the third row 1808 may be associated with a third alignment block1828, the fourth row 1810 may be associated with a fourth alignmentblock 1830, and the fifth row 1812 may be associated with a fifthalignment block 1832. To illustrate, the first alignment block 1824 maybe adapted to collect image data lines from each camera in the first row1804 (i.e., five cameras in the five columns 1814-1822 of the array1802). The first alignment block 1824 may be adapted to align the imagedata lines in the same order and send the image data lines forprocessing as a single line. The first alignment block 1824 may beadapted to send the image data lines for processing as a single line toa first VFE component 1834 to be processed for color such as describedwith respect to the combiner of 206 of FIGS. 2, 3, and 5. A size of thesingle line may be determined based on the number of cameras (i.e., fivecameras) and the horizontal size (i.e., “H_res” as described in FIG. 15)of each camera in the first row 1804.

The second alignment block 1826 may be adapted to collect images fromeach camera in the second row 1806, to align the images in a same order,and to send the images for processing as a single line to a second VFEcomponent 1836. The third alignment block 1828 may be adapted to collectan image data line from each camera in the third row 1808, to align theimage data lines in a same order, and to send the image data lines forprocessing as a single line to a third VFE component 1838. The fourthalignment block 1830 may be adapted to collect image data lines fromeach camera in the fourth row 1810, to align the image data lines in asame order, and to send the image data lines for processing as a singleline to a fourth VFE component 1840. The fifth alignment block 1832 maybe adapted to collect image data lines from each camera in the fifth row1812, to align the image data lines in a same order, and to send theimage data lines for processing as a single line to a fifth VFEcomponent 1842.

A control synchronization block 1844 may be used to synchronize each ofthe cameras of the array 1802 (i.e., twenty five cameras in the case ofthe illustrative five-by-five array 1802 of FIG. 18) in a manner similarto the sensor synchronizer 230. In a particular embodiment, the controlsynchronization block 1834 may implement the control interface 1704 ofFIG. 17. The control synchronization block 1844 may be communicativelycoupled to each of the cameras of the array 1802 and to each of the VFEcomponents 1834-1842. Synchronization of all cameras in the array 1802may allow for usage of a rolling shutter on a high resolution. As allcameras may be read out at the same time, the rolling shutter effect maybe diminished (with the size of the array). For example, in thefive-by-five array 1802 of FIG. 18, synchronization of the twenty fivecameras may diminish the rolling shutter effect associated with aComplementary Metal Oxide Semiconductor (CMOS) camera.

The first VFE component 1834 may be communicatively coupled to a firstGPU component 1846 to align and rectify each individual image in thefirst row 1804 (i.e., five images captured by the cameras in the fivecolumns 1814-1822). The second VFE component 1836 may be communicativelycoupled to a second GPU component 1848 to align and rectify eachindividual image in the second row 1806. The third VFE component 1838may be communicatively coupled to a third GPU component 1850 to alignand rectify each individual image in the third row 1808. The fourth VFEcomponent 1840 may be communicatively coupled to a fourth GPU component1852 to align and rectify each individual image in the fourth row 1810.The fifth VFE component 1842 may be communicatively coupled to a fifthGPU component 1854 to align and rectify each individual image in thefifth row 1812. Each of the GPU components 1846-1854 may becommunicatively coupled to a GPU processing component 1856 that isadapted to combine all of the rows 1804-1812 together, resulting in afinal image.

In the embodiment illustrated in FIG. 18, each of the alignment blocks1824-1832 is associated with its own VFE component and its own GPUrectification and alignment component. Thus, FIG. 18 illustrates thateach of the rows 1804-1812 may be processed in parallel for color usingseparate VFE components, and each of the rows 1804-1812 may be processedusing separate GPU components to align and rectify each individual imagein a particular row. Alternatively, each of the alignment blocks1824-1832 may be associated with a single VFE component and a single GPUrectification and alignment component (see FIG. 19).

Referring to FIG. 19, a particular embodiment of a camera arrayprocessing system is illustrated and generally designated 1900. Allcameras in an array 1902 may be synchronized, aligned, and processedprior to being combined into a final image. In the embodimentillustrated in FIG. 19, the array 1902 includes a five-by-five array ofcameras. Alternatively, the array 1902 may include any other number ofcameras arranged in a two-dimensional array. FIG. 19 illustrates that asingle VFE component and a single GPU rectification and alignmentcomponent may be used to process all rows of the array 1902, rather thanthe multiple VFE and GPU rectification and alignment componentsillustrated in FIG. 18.

The five-by-five array 1902 illustrated in FIG. 19 includes a first row1904, a second row 1906, a third row 1908, a fourth row 1910, and afifth row 1912. Further, the five-by-five array 1902 includes a firstcolumn 1914, a second column 1916, a third column 1918, a fourth column1920, and a fifth column 1922. The first row 1904 may be associated witha first alignment block 1924, the second row 1906 may be associated witha second alignment block 1926, the third row 1908 may be associated witha third alignment block 1928, the fourth row 1910 may be associated witha fourth alignment block 1930, and the fifth row 1912 may be associatedwith a fifth alignment block 1932. To illustrate, the first alignmentblock 1924 may be adapted to collect image data lines from each camerain the first row 1904 (i.e., five cameras in the five columns 1914-1922of the array 1902).

The first alignment block 1924 may be adapted to align the image datalines in a same order and send the image data lines for processing as asingle line. The second alignment block 1926 may be adapted to collectimage data lines from each camera in the second row 1906, to align theimage data lines in a same order, and to send the image data lines forprocessing as a single line. The third alignment block 1928 may beadapted to collect image data lines from each camera in the third row1908, to align the image data lines in a same order, and to send theimage data lines for processing as a single line. The fourth alignmentblock 1930 may be adapted to collect image data lines from each camerain the fourth row 1910, to align the image data lines in a same order,and to send the image data lines for processing as a single line. Thefifth alignment block 1932 may be adapted to collect image data linesfrom each camera in the fifth row 1912, to align the image data lines ina same order, and to send the image data lines for processing as asingle line.

In the embodiment illustrated in FIG. 19, each of the alignment blocks1924-1932 may be adapted to send its images for processing to a singleVFE component 1934 to be processed for color. The single VFE component1934 may process each of the five lines that are sent from the fivealignment blocks 1924-1932. As noted above, the size of a single linefrom a particular alignment block may be determined based on a number ofcameras in a particular row (i.e., five cameras) and a horizontal size(i.e., “H_res” as described in FIG. 15) of each camera in the particularrow. As such, the size of the multiple lines processed by the single VFEcomponent 1934 of FIG. 19 may be five times the size of a single lineprocessed by one of the VFE components 1834-1842 of FIG. 18.

A control synchronization block 1936 may be used to synchronize each ofthe cameras of the array 1902 such as providing common control signalsto cameras in one or more rows 1904-1912. The control synchronizationblock 1936 may be communicatively coupled to each of the cameras of thearray 1902 and to the single VFE components 1934. Synchronization of allcameras in the array 1902 may allow for usage of a rolling shutter on ahigh resolution. As all cameras may be read out at the same time, arolling shutter effect may be diminished (with the size of the array).For example, in the five-by-five array 1902 of FIG. 19, synchronizationof the twenty five cameras may diminish a rolling shutter effectassociated with a CMOS camera.

The single VFE component 1934 may be communicatively coupled to a singleGPU component 1938 to align and rectify each individual image in each ofthe rows 1904-1912. As such, the single GPU component 1938 of FIG. 19may align and rectify twenty five images compared to the five imagesprocessed by each of the GPU alignment and rectification components1846-1854 of FIG. 18. The single GPU component 1938 may becommunicatively coupled to a GPU processing component 1940 that isadapted to combine all of the rows 1904-1912 together, resulting in afinal image.

FIG. 20 illustrates a high-resolution digital camera system 2000 thatincludes a main lens 2002 configured to focus incoming light 2004 andmultiple cameras arranged in an array 2006. A high-resolution image canbe generated as a composite (or “mosaic”) image by combining imagescaptured at each of the cameras in the array 2006. For example, each ofthe cameras of the array 2006 may be a CMOS-type camera or a ChargeCoupled Device (CCD) type camera. The main lens 2002 may focus acaptured scene to a plane 2008 (referred to as a “focus plane” of themain lens 2002 or an “object plane” of cameras in the array 2006), andeach camera in the array 2006 may capture a portion of the total image.Each camera of the array 2006 has a field of view that partiallyoverlaps its neighbors' fields of view at the plane 2008. This overlapmay enable images taken from neighboring cameras in the array 2006 to bealigned on a row-by-row basis and “stitched” together during subsequentprocessing and may provide tolerance for non-ideal position andalignment of cameras within the array (such as described with respect toFIG. 16).

A composite image can be generated by aligning image data from thecameras of the array 2006 on a row-by-row basis. For example, the array2006 of FIG. 20 includes a three-by-three array with three rows. Eachcamera within a particular row of the array 2006 (an “array row”) mayinclude an image sensor that has light detectors arranged in columns androws (“sensor columns” and “sensor rows”). The cameras within an arrayrow may be positioned so that sensor rows are substantially aligned. Toillustrate, a first sensor row of each camera in an array row issubstantially aligned with the first sensor row of every other camera inthe same array row.

During image capture, the first sensor row of image data is read fromeach camera in an array row and provided to image processing circuitry(such as described with respect to FIGS. 17-19). The image data from thefirst sensor row is merged according to the position of each camera inthe array row. The merged image data is processed as if it were a singlerow of image data from a larger camera. The second, third, andsubsequent image sensor rows are read, merged, and provided to the imageprocessing circuitry to be processed as rows of the composite image.Each array row may be processed in parallel with the other array rows.

The arrangement of FIG. 20 may provide an inexpensive alternative tohigh-resolution cameras. For example, a 100 megapixel (mpix) camera canbe built using an array of twenty 5 mpix CMOS cameras behind a mainlens. Because image capture can be performed using multiple CMOS cameraswith each camera capturing a portion of the image, “rolling shutter”artifacts may be reduced as compared to a single 100 mpix CMOS cameracapturing the entire image.

Referring to FIG. 21, a particular embodiment of an implementation of amulti-camera module is depicted and generally designated 2100. Thesystem 2100 illustrates a multiple camera module, such as a multiplecamera module as illustrated in FIGS. 1-5, mounted on an automobile. Themultiple camera module may be configured to generate synchronized linedata frames for formatting as three dimensional image or video data,such as described with respect to FIGS. 6-7. By mounting the multiplecamera module to a rear portion of the automobile, a three dimensionalview may be obtained to provide an operator of the automobile with depthperception on an internal display (not shown) while backing theautomobile. It will be understood that the multiple camera module may bemounted to any type of vehicle, without limitation to automobiles.

Referring to FIG. 22, a flow diagram of a particular illustrativeembodiment of a method of combining data from multiple sensors into asynchronized data line depicted and generally designated 2200. As anillustrative example, the method 2200 may be performed by the system ofFIG. 2, the system of FIG. 3, the system of FIG. 5, or any combinationthereof

A common control signal may be provided to multiple image sensors to besynchronized, at 2202. For example, the common control signal mayinclude a common clock signal and a common reset signal, such as thecommon control signals 404-410 depicted in FIG. 4.

A first data line from a first image sensor of the multiple imagesensors may be received, at 2204. A second data line from a second imagesensor of the multiple image sensors may be received, at 2206. Forexample, the first sensor and the second sensor may be the sensors 202,204 of FIG. 2.

The first data line and the second data line may be combined line togenerate a synchronized data line, at 2208. For example, the method 2200may include interleaving a first data stream received from the firstimage sensor and a second data stream received from the second imagesensor on a line by line basis. The synchronized data line may be formedas described with respect to the combiner 406 of combining the firstsensor image data 422 and the second sensor image data 432 illustratedin FIG. 5.

The synchronized data line may form part of a frame, such as the frame660 of FIG. 6. The frame can include a first section (e.g. the firstsection 652) including first image data from the first image sensor, asecond section (e.g. the second section 654) including second image datafrom the second image sensor, and a gap section (e.g. the gap section656) including non-image data disposed between the first section and thesecond section. In other embodiments, the frame may not include a gapsection between the first section and the second section.

Receiving the first data line may be completed before receiving thesecond data line is completed, and the synchronized data line may begenerated after receiving the second data line is completed. As anexample, the combined data line 822 of FIG. 8 is generated after thesecond sensor first line data 812 has been received.

In an embodiment having more than two image sensors, a third data linemay be received from a third image sensor of the multiple image sensors,such as illustrated in FIG. 11. The third data line can be combined withthe first data line and the second data line to generate thesynchronized data line, such as the first synchronized data line 1150 ofFIG. 11.

Referring to FIG. 23, a flow diagram of a particular illustrativeembodiment of a method of providing a common control signal to multipleimage sensors and providing a synchronized data line to an imageprocessor via a single camera input of the image processor is depictedand generally designated 2300. The method 2300 may be performed at oneor more of the systems of FIGS. 2, 3, and 5, as illustrative,non-limiting examples.

A common control signal may be provided to multiple image sensors, at2302. Each of the multiple image sensors may be responsive to the commoncontrol signal to generate image data. For example, the common controlsignal may be provided by a sensor synchronizer that is coupled to eachof the multiple image sensors, such as the sensor synchronizer 230 ofFIG. 2. To illustrate, the sensor synchronizer may be coupled to each ofthe multiple image sensors via an inter-integrated circuit (I2C) controlinterface, via an interface compliant with a camera serial interface(CSI) specification, or via an interface compliant with a cameraparallel interface (CPI) specification.

Synchronized data output from each of the multiple image sensors may bereceived, at 2304. A first data line may be received from a first imagesensor of the multiple image sensors and a second data line may bereceived from a second image sensor of the multiple image sensors.Receiving the first data line may be completed before receiving thesecond data line is completed, and a synchronized data line may begenerated after receiving the second data line is completed, such as thecombined data line 822 that is generated after the second sensor firstline data 812 has been received in FIG. 8.

The synchronized data output from each of the multiple image sensors maybe combined to generate a synchronized data line, at 2306. For example,the combiner 206 of FIG. 2 may interleaving a first data stream receivedfrom the first image sensor 202 and a second data stream received fromthe second image sensor 204 on a line by line basis.

The synchronized data line may be provided to an image processor via asingle camera input of the image processor, at 2308. The synchronizeddata line may form part of a frame that has a multiple rows, such as theframe 660 of FIG. 6. For example, the frame may include a first sectionincluding first image data from the first image sensor, a second sectionincluding second image data from the second image sensor, and a gapsection between the first and second sections.

Referring to FIG. 24, a flow diagram of a particular illustrativeembodiment of a method of providing a common control signal to multipleimage sensors and receiving synchronized data output from each of themultiple image sensors is depicted and generally designated 2400.

A common control signal to multiple image sensors may be provided, at2402. Each of the multiple image sensors is responsive to the commoncontrol signal to generate image data. For example, the common controlsignal may provided by a sensor synchronizer that is coupled to each ofthe multiple image sensors, such as the sensor synchronizer 230 of anyof FIGS. 2-5, the control synchronization block 1844 of FIG. 18, thecontrol synchronization block 1936 of FIG. 19, or any combinationthereof.

Synchronized data output from each of the multiple image sensors may bereceived, at 2404. The synchronized data output may include first datalines received from a first image sensor and second data lines receivedfrom a second image sensor. A phase offset between each received dataline from the first image sensor and each corresponding data line fromthe second image sensor may be substantially constant, such as theone-line phase difference of FIG. 9, the two-line phase difference ofFIG. 8, or the 3-line phase difference of FIG. 12, as illustrative,non-limiting examples.

Referring to FIG. 25, a flow diagram of a particular illustrativeembodiment of a method of receiving a common control signal at multipleimage sensors and generating synchronized data output from each of themultiple image sensors is depicted and generally designated 2500.

A common control signal may be received at multiple image sensors, at2502. Each of the multiple image sensors is responsive to the commoncontrol signal to generate image data. For example, the common controlsignal may received from a sensor synchronizer that is coupled to eachof the multiple image sensors, such as the sensor synchronizer 230 ofany of FIGS. 2-5, the control synchronization block 1844 of FIG. 18, thecontrol synchronization block 1936 of FIG. 19, or any combinationthereof.

Synchronized data output from each of the multiple image sensors may begenerated, at 2504. The synchronized data output may include first datalines received from a first image sensor and second data lines receivedfrom a second image sensor. A phase offset between each received dataline from the first image sensor and each corresponding data line fromthe second image sensor may be substantially constant, such as theone-line phase difference of FIG. 9, the two-line phase difference ofFIG. 8, or the 3-line phase difference of FIG. 12, as illustrative,non-limiting examples.

Referring to FIG. 26, a flow diagram of a particular illustrativeembodiment of a method of combining data from multiple sensors at animage signal processor having an input for a single camera is depictedand generally designated 2600.

Lines of image data may be received at an image processor having aninput for a single camera, at 2602. Each line of the image data mayinclude first line data from a first image captured by a first cameraand second line data from a second image captured by a second camera. Asan illustrative, non-limiting example, the image processor may includethe image signal processor 208 of FIGS. 2-3 or FIG. 5, the imageprocessor 1308 of FIG. 13, the VFEs 1834-1842 of FIG. 18, the VFEs1934-1942 of FIG. 19, or any combination thereof.

The lines of image data may be received at the image processor from acombiner that is coupled to the first camera and to the second camera.Line by line readout of first image data from the first camera andsecond image data from the second camera may be synchronized, using thecombiner, to generate each line of the image data. For example, thecombiner may be the combiner 206 of FIGS. 2-3 or FIG. 5, the combiner1306 of FIG. 13, the data alignment blocks 1824-1832 of FIG. 18, thedata alignment blocks 1924-1932 of FIG. 19, or any combination thereof.

An output frame having a first section corresponding to line data of thefirst image and having a second section corresponding to line data ofthe second image may be generated, at 2604. The first section and thesecond section may be configured to be used to generate athree-dimensional (3D) image format or a 3D video format.

In a particular embodiment, the output frame is processed to generate 3Dimage data, and the 3D image data is sent to a display device. Inanother embodiment, the output frame is processed to generate 3D videodata, and the 3D video data is sent to a display device, such as thedisplay device 1310 of FIG. 13. To illustrate, the display device may bea component of at least one of a communication device, a camera, apersonal digital assistant, and a computer.

Referring to FIG. 27, a flow diagram of an illustrative embodiment of amethod of combining data from multiple sensors into a frame is depictedand generally designated 2700. As an illustrative example, the method2700 may be performed by the system of FIG. 2, the system of FIG. 5, orany combination thereof.

A first data stream may be received from a first image sensor, such asthe first sensor 202 of FIG. 2, at 2702. The first data stream, such asthe first image data stream 212 of FIG. 2, the first data stream 602 ofFIG. 6, or the timing data signal 420 and the image data signal 422 ofFIG. 5, may correspond to first image data of an image. A second datastream may be received from a second image sensor, such as the secondsensor 204 of FIG. 2, at 2704. The second data stream, such as thesecond image data stream 214 of FIG. 2, the second data stream 604 ofFIG. 6, or the timing signal data 430 and the image data signal 432 ofFIG. 5, may correspond to second image data of the image.

Data from the first data stream and data from the second data stream maybe combined, at 2706. For example, a combiner, such as the combiner 206of FIG. 2 or the combiner 206 of FIG. 5, may combine the first imagedata from the first data stream and the second image data from thesecond data stream and generate a resulting frame. To illustrate, thefirst data stream may include data associated with a first line of thefirst image data including line data having a first line index value,line data having a second line index value, etc. The second data streammay include line data that corresponds to that of the first data stream,including corresponding line data having a first line index value, andcorresponding line data having a second line index value, etc. The linedata from the first data stream having the first line index value andthe line data from the second data stream having the corresponding firstline index value may be appended to each other, or combined, to form asingle image line. The process may be repeated for each line index valueto generate the resulting frame, such as the frame 660 of FIG. 6. In aparticular embodiment, the frame may include a plurality of rows, whereeach row corresponds to a line index value and stores a line of thefirst image data having the line index value and stores a correspondingline of the second image data having the line index value. In aparticular embodiment, when the first line of the first image data andthe corresponding line of the second image data are combined into asingle image line, the size of the single image line is substantiallydouble that of the first line of the first image data or thecorresponding line of the second image data.

The frame may be processed at an image signal processor to generate aprocessed frame, at 2708. In a particular embodiment, the image signalprocessor may be the image signal processor 208 of FIG. 2 or the imagesignal processor 208 of FIG. 5, and the processed frame may be theprocessed frame 240 of FIG. 2 or the processed frame 650 of FIG. 6. In aparticular embodiment, the processed frame may include a first sectionincluding first image data from the first image sensor, such as thefirst section 652, a second section including second image data from thesecond image sensor, such as the second section 654, and a gap section,such as the gap section 656 of FIG. 6. The gap section may includenon-image data disposed between the first section and the secondsection. In a particular embodiment, the first section may include aline of the first image data and the second section may include acorresponding line of the second image data.

The processed frame may be output to be displayed at a display device,at 2710. In a particular embodiment, the first image sensor and thesecond image sensor are each directly responsive to the image signalprocessor, and the display device may be decoupled from the image signalprocessor.

Referring to FIG. 28, a flow diagram of an illustrative embodiment of amethod of synchronizing a first image sensor and a second image sensoris depicted and generally designated 2800. As an illustrative example,the method 2800 may be performed at the system 200 of FIG. 2, the system600 of FIG. 5, or any combination thereof.

First image data of an image may be received from a first image sensor,at 2802. In a particular embodiment, the first image sensor may be thefirst sensor 202 of FIG. 2. A first data stream associated with thefirst image data may be received from the first image sensor, at 2804.In a particular embodiment, the first data stream may be generated bythe image sensor and may be the first image data stream 212 of FIG. 2,the first data stream 602 of FIG. 6, or the timing data signal 420 andthe image data signal 422 of FIG. 5.

Second image data of the image may be received from a second imagesensor, at 2806. In a particular embodiment, the second image sensor maybe the second sensor 204 of FIG. 2. A second data stream associated withthe second image data may be received from the second image sensor, at2808. In a particular embodiment, the second data stream may begenerated by the image sensor and may be the second image data stream214 of FIG. 2, the second data stream 604 of FIG. 6, or the timing datasignal 430 and the image data signal 432 of FIG. 5.

Line by line exposure of the first image sensor and the second imagesensor during image data acquisition may be synchronized, at 2810. In aparticular embodiment, the synchronization may occur during image dataacquisition of an image at a host including a combiner, such as thecombiner 206 of FIG. 2 or the combiner 206 of FIG. 5. In a particularembodiment, the first image sensor and the second image sensor areindependent of each other. For example, the first and second sensors202, 204 of FIG. 2 are directly responsive to the image signal processor208 via the control signal 232 to have similar timing characteristicswhile remaining independent of each other. The first data stream and thesecond data stream may be interleaved, at 2812. In a particularembodiment, the first data stream and the second data stream may beinterleaved on a line by line basis. For example, line data from thefirst data stream having a first line index value and line data from thesecond data stream having a corresponding first line index value may beappended to each other to form an interleaved single image line.

Thus, the combined data may be efficiently processed using a singleimage signal processor. Thus, overall image system cost and complexitymay be reduced compared to multiple processor systems in which aprocessor is assigned to each sensor.

Referring to FIG. 29, a flow diagram of a first illustrative embodimentof a method of combining data from multiple image sensors to generate 3Dimage data is depicted and generally designated 2900. As an illustrativeexample, the method 2900 may be performed by the system of FIG. 13.

The method includes synchronizing line by line readout of first imagedata from a first camera and a second camera to generate rows of imagedata, at 2902. The first image data from the first camera may be theimage data stream 1314 from the first image sensor 1302 of FIG. 1 andthe second image data may be the image data stream 1316 from the secondimage sensor 1304 of FIG. 13.

The method includes receiving rows of the image data at an imageprocessor having an input for a single camera, at 2904. Each row of theimage data includes data from a row of a first image captured by thefirst camera and data from a row of a second image captured by thesecond camera. The rows of image data may be the data out stream 706depicted in FIG. 7.

The method includes generating, with the image processor, an outputhaving one of a 3D image format and a 3D video format, at 2906. Theoutput corresponds to the first image and the second image. The outputis sent to a display device (e.g., the display device 1310 of FIG. 13),at 2908.

Referring to FIG. 30, a flow diagram of an illustrative embodiment of amethod of combining data from multiple sensors into a frame is depictedand generally designated 3000. As an illustrative example, the method3000 may be performed by the system of FIG. 13.

A first data stream is received from a first image sensor, such as thefirst image sensor 1302 of FIG. 13, at 3002. The first data stream, suchas the first image data stream 1314 of FIG. 13 or the first data stream702 of FIG. 7 may correspond to first image data of a first image. Asecond data stream may be received from a second image sensor, such asthe second image sensor 1304 of FIG. 13, at 3004. The second datastream, such as the second image data stream 1316 of FIG. 13, the seconddata stream 704 of FIG. 7 may correspond to second image data of asecond image. The first image and the second image may be images of ascene. The first image and the second image of the scene may be taken atsubstantially the same time, or may be taken at different times. Thefirst image may be taken from a different vantage point than the secondimage so that depth information can be determined from the first imageand the second image of the scene.

Data from the first data stream and data from the second data stream iscombined, at 3006. For example, a combiner, such as the combiner 1306 ofFIG. 13 may combine the first image data from the first data stream andthe second image data from the second data stream and generate aresulting frame. To illustrate, the first data stream may include dataassociated with a first line of the first image data including line datahaving a first line index value, line data having a second line indexvalue, etc. The second data stream may include line data thatcorresponds to that of the first data stream, including correspondingline data having a first line index value, and corresponding line datahaving a second line index value, etc. The line data from the first datastream having the first line index value and the line data from thesecond data stream having the corresponding first line index value maybe appended to each other, or combined, to form a single image line. Theprocess may be repeated for each line index value to generate theresulting frame, such as the frame 740 of FIG. 7. In a particularembodiment, the frame may include a plurality of rows, where each rowcorresponds to a line index value and stores a line of the first imagedata having the line index value and stores a corresponding line of thesecond image data having the line index value. In a particularembodiment, when the first line of the first image data and thecorresponding line of the second image data are combined into a singleimage line, the size of the single image line is substantially doublethat of the first line of the first image data or the corresponding lineof the second image data.

The frame is received as rows of image data via an input for a singlecamera, at 3008. In a particular embodiment, the input for the singlecamera may be the input of an image processor, such as the imageprocessor 1308 of FIG. 13. The frame may be frame 740 of FIG. 7.

An output is generated from the frame, at 3010. The output has one of a3D image format and a 3D video format. The output corresponds to thefirst image and the second image. The output may be the processed framedata 1320 of FIG. 13. The output is sent to a display device, at 3012.In a particular embodiment, the display device may be the display device1310 of FIG. 13.

Referring to FIG. 31, a flow diagram of an illustrative embodiment of amethod of synchronizing a first image sensor and a second image sensoris depicted and generally designated 3100. As an illustrative example,the method 3100 may be performed by the system 1300 of FIG. 13.

First image data of an image may be received from a first image sensor,at 3102. In a particular embodiment, the first image sensor may be thefirst image sensor 1302 of FIG. 13. A first data stream associated withthe first image data may be received from the first image sensor, at3104. In a particular embodiment, the first data stream may be generatedby the image sensor and may be the first image data stream 1314 of FIG.13 or the first data stream 702 of FIG. 7.

Second image data of the image may be received from a second imagesensor, at 3106. In a particular embodiment, the second image sensor maybe the second image sensor 1304 of FIG. 13. A second data streamassociated with the second image data may be received from the secondimage sensor, at 3108. In a particular embodiment, the second datastream may be generated by the image sensor and may be the second imagedata stream 1316 of FIG. 13 or the second data stream 704 of FIG. 7.

Line by line exposure of the first image sensor and the second imagesensor during image data acquisition may be synchronized, at 3110. In aparticular embodiment, the synchronization may occur during image dataacquisition of an image at a host including a combiner, such as thecombiner 1306 of FIG. 13. In a particular embodiment, the first imagesensor and the second image sensor are independent of each other. Forexample, the first and second image sensors 1302, 1304 of FIG. 13 may bedirectly responsive to the image processor 1308 via a control signal tohave similar timing characteristics while remaining independent of eachother. In another embodiment, the first and second image sensors 1302,1304 are directly responsive to the combiner 1306 via a control signalto have similar timing characteristics while remaining independent ofeach other. The first data stream and the second data stream may beinterleaved, at 3112. In a particular embodiment, the first data streamand the second data stream may be interleaved on a line by line basis.For example, line data from the first data stream having a first lineindex value and line data from the second data stream having acorresponding first line index value may be appended to each other toform an interleaved single image line.

Thus, the combined data may be efficiently processed using a singleimage processor. Thus, overall image system cost and complexity may bereduced compared to multiple processor systems in which a processor isassigned to each sensor.

Referring to FIG. 32, a flow diagram of an illustrative embodiment of amethod of generating a 3D image with an image processor is depicted andgenerally designated as 3200. The image processor may be the imageprocessor 1308 of FIG. 13. The method 3200 may be used when the imageprocessor treats a frame received from a combiner (e.g., the combiner1306 of FIG. 1), a frame from a memory, or when a user chooses to altera displayed 3D image using a zoom feature or a pan feature of a devicedisplaying the 3D image.

The image processor rectifies a first image and a second image based onparameters of a calibration matrix, at 3202. The calibration matrix mayprovide adjustments for relative positions of a first image sensor and asecond image sensor that capture the first image and the second image.The relative positions of the two cameras may be selected to ensureminimal scene distortion and eye strain. The calibration matrix may bedetermined during a manufacturing process for a device that takes the 3Dimage where the positions of the first image sensor and the second imagesensor are fixed relative to each other. The calibration may be storedin a memory of the device. For a device that takes the 3D image wherethe positions of the first image sensor, the second image sensor, orboth are adjustable, a processor of the device may be used to run acalibration routine to determine the calibration matrix and store thecalibration matrix in the memory. The calibration routine may requirethe first image sensor and the second image sensor to be focused on aparticular calibration scene positioned a set distance from the imagesensors. The calibration routine may be performed after positionadjustment of the image sensors relative to each other.

The image processor detects keypoints in the first image, at 3204. Theimage processor may detect distinctive (high frequency) points in thefirst image. The image processor block matches between local imagepatches in the first image and the second image to compute disparitiesfor each detected keypoint in the first image, at 3206. A reliabilityestimator may be produced for every keypoint to insure that erroneousmatches are discarded. The image processor determines a convergenceadjustment based on a disparity range determined from the computeddisparities, at 3208. The convergence adjustment takes scene depth anddisplay geometry into consideration.

The image processor selectively shifts at least one of the first imageand the second image based on the convergence adjustment when theconvergence adjustment is within capabilities of a display device thatwill display the 3D image to generate output, at 3210. The imageprocessor uses the first image with disparity adjusted to match amajority of the scene when the convergence adjustment is not within thecapabilities of the display device to generate the output, at 3212. Theimage processor crops the output based on one or more displaycharacteristics of the display device, at 3214.

FIG. 33 is a block diagram of a particular embodiment of an imageprocessing system 3300 to combine data from multiple image sensors. Theimage processing system 3300 may include an image sensor device 3302that is coupled to a first lens 3304, coupled to a second lens 3306, andcoupled to an application processor chipset of a portable multimediadevice 3308. The image sensor device 3302 may include a combiner 3310and an image processor 3312 that receives input for a single camera. Theimage processor 3312 may receive the single camera input from thecombiner 3310 or from a memory device 3314 of the application processorchipset of the portable multimedia device 3308. The combiner 3310 maycombine data from a first data stream and from a second data stream togenerate a frame, such as by implementing the system 1300 of FIG. 13, byoperating in accordance with any of the embodiments of FIGS. 29-31, orany combination thereof.

The combiner 3310 is coupled to receive image data from a first sensor3316 via a first analog-to-digital convertor 3318. The combiner 3310 iscoupled to receive image data from a second sensor 3320 via a secondanalog-to-digital convertor 3322. The combiner 3310 or the imageprocessor 3312 may control the first sensor 3316 and the second sensor3320, which may be otherwise independent of each other. In a particularembodiment, the image processor 3312 may control the first sensor 3316and the second sensor 3320 via a sensor synchronizer 3330 (shown inshadow).

In a particular embodiment, an integrated circuit that includes imageprocessing circuitry, such as the combiner 3310, is configured togenerate a frame. The image processing circuitry is configured toreceive a first data stream from a first image sensor, such as the firstsensor 3316, to receive a second data stream from a second image sensor,such as the second sensor 3320, and to combine data from the first datastream and from the second data stream to generate the frame. Forexample, the first data stream 702 and the second data stream 704 ofFIG. 7 may be combined by the combiner 3310 to form the frame 740 ofFIG. 7.

Output from the combiner 3310 may be sent to a memory device 3314 of theapplication processor chipset of the portable multimedia device 3308, toan image processor 3312, or both. The image processor 3312 may beconfigured to perform additional image processing operations, such asone or more operations performed by an image processing system. Theimage processor 3312 may receive a frame from the combiner 3310 or fromthe memory device 3314. The image processor 3312 may produce processedimage data such as a processed frame having a 3D image format or a 3Dvideo format. In an embodiment, an average time for producing processedimage data is about 20 milliseconds. The image processor 3312 mayprovide the processed image data to the application processor chipset ofthe portable multimedia device 3308 for further processing,transmission, storage, display to a display device 3324, or anycombination thereof.

Referring to FIG. 34, a block diagram of a particular illustrativeembodiment of an electronic device, such as a wireless phone, includinga frame generator module, as described herein, is depicted and generallydesignated 3400. The device 3400 includes a processor 3410 coupled to amemory 3432. The processor includes or is coupled to a controller 3464.Alternatively, the electronic device may be a set top box, a musicplayer, a video player, an entertainment unit, a navigation device, acommunications device, a personal digital assistant (PDA), a camera, afixed location data unit, or a computer.

FIG. 34 also shows a display controller 3426 that is coupled to theprocessor 3410 and to a display 3428. A coder/decoder (CODEC) 3434 canalso be coupled to the processor 3410. A speaker 3439 and a microphone3438 can be coupled to the CODEC 3434. A camera controller 3470 can alsobe coupled to the processor 3410. A first camera 3472 and a secondcamera 3473 can be coupled to the camera controller 3470.

FIG. 34 also indicates that a wireless interface 3440 can be coupled tothe processor 3410 and to a wireless antenna 3442. In a particularembodiment, the processor 3410, the display controller 3426, the memory3432, the CODEC 3434, the wireless interface 3440, and the controller3464 are included in a system-in-package or system-on-chip 3422. In aparticular embodiment, an input device 3430 and a power supply 3444 arecoupled to the on-chip system 3422. Moreover, in a particularembodiment, as illustrated in FIG. 34, the display 3428, the inputdevice 3430, the speaker 3439, the microphone 3438, the wireless antenna3442, and the power supply 3444 are external to the on-chip system 3422.However, each can be coupled to a component of the on-chip system 3422,such as an interface or a controller.

In a particular embodiment, the processor 3410 executesprocessor-readable program instructions from a processor-readablemedium, such as program instructions 3482 stored at the memory 3432. Forexample, the memory 3432 may be readable by the processor 3410 and theinstructions 3482 may be operational instructions that are executable bythe processor 3410 to perform the method 2200 of FIG. 22. For example,the instructions 3482 may include instructions that are executable bythe processor 3410 to receive a first data stream from a first imagesensor, such as the camera 3473 to receive a second data stream from asecond image sensor, such as the camera 3472, and to combine data fromthe first data stream and from the second data stream to generate aframe. For example, the first image sensor may be the first sensor 202of FIG. 2 and the second image sensor may be the second sensor 204 ofFIG. 2. The instructions 3482 may further include instructions that areexecutable by the processor 3410 to process the frame at the processor3410 or at an image signal processor (not shown) to generate a processedframe. The instructions 3482 may further include instructions that areexecutable by the processor 3410 to output the processed frame to bedisplayed at the display device 3428 or stored at the memory 3432 asimage data 3480.

Referring to FIG. 35, a block diagram of a particular illustrativeembodiment of an electronic device, such as a mobile phone, is depictedand generally designated 3500. The device 3500 includes a processor 3502coupled to a memory 3504. The processor 3502 includes or is coupled toan image processor 3506. The image processor 3506 may receive a singlecamera input and may output 3D data 3590. The 3D data 3590 may be in 3Dimage format or 3D video format. Alternatively, the electronic device3500 may be a set top box, a music player, a video player, anentertainment unit, a navigation device, a communications device, apersonal digital assistant (PDA), a camera, a fixed location data unit,a computer, or combinations thereof.

FIG. 35 also shows a display controller 3508 that is coupled to theprocessor 3502 and to a display 3510. A coder/decoder (CODEC) 3512 canalso be coupled to the processor 3502. A speaker 3514 and a microphone3516 can be coupled to the CODEC 3512. A camera controller 3518 can alsobe coupled to the processor 3502. The camera controller 3518 may includea combiner 3520. The combiner 3520 may provide image data to the imageprocessor 3506. As illustrative examples, the combiner 3520 may be thecombiner 1306 of FIG. 13, or other hardware circuitry or processorconfigured to combine data from multiple cameras as illustrated withrespect to FIG. 7. A first camera 3522 and a second camera 3524 can becoupled to the camera controller 3518.

FIG. 35 also indicates that a wireless interface 3526 can be coupled tothe processor 3502 and to a wireless antenna 3528. In a particularembodiment, the processor 3502, the display controller 3508, the memory3504, the CODEC 3512, the camera controller 3518, and the wirelessinterface 3526 are included in a system-in-package or system-on-chip3530. In a particular embodiment, an input device 3532 and a powersupply 3534 are coupled to the on-chip system 3530. Moreover, in aparticular embodiment, as illustrated in FIG. 35, the display 3510, theinput device 3532, the speaker 3514, the microphone 3516, the wirelessantenna 3528, and the power supply 3534 are external to the on-chipsystem 3530. However, each can be coupled to a component of the on-chipsystem 3530, such as an interface or a controller.

In a particular embodiment, the processor 3502 executesprocessor-readable program instructions from a processor-readablemedium, such as program instructions 3536 stored at the memory 3504. Forexample, the memory 3504 may be readable by the processor 3502 and theinstructions 3536 may be operational instructions that are executable bythe processor 3502 to perform the method 2500 of FIG. 25. For example,the instructions 3536 may include instructions that are executable bythe processor 3502 to receive a first data stream from a first imagesensor, such as the camera 3522, to receive a second data stream from asecond image sensor, such as the camera 3524, and to combine data fromthe first data stream and from the second data stream using the combiner3520 of the camera controller 3518 to generate a frame. For example, thefirst image sensor may be the first image sensor 1302 of FIG. 13, andthe second image sensor may be the second image sensor 1304 of FIG. 13.The instructions 3536 may further include instructions that areexecutable by the processor 3502 to process the frame at the imageprocessor 3506 to generate a processed frame. The instructions 3536 mayfurther include instructions that are executable by the processor 3502to output the processed frame as 3D data to the display controller 3508for display at the display device 3510 or to store the processed frameat the memory 3504 as image data 3538.

Those of skill would further appreciate that the various illustrativelogical blocks, configurations, modules, circuits, and algorithm stepsdescribed in connection with the embodiments disclosed herein may beimplemented as electronic hardware, computer software executed by aprocessor, or combinations of both. Various illustrative components,blocks, configurations, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software executed by aprocessor depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans may implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the present disclosure.

The methods of FIGS. 22-32 may be performed by executing program codethat may be stored in memory in the form of computer readableinstructions. In that case, a processor, such as a digital signalprocessor (DSP) an image signal processor (ISP), or other processor, mayexecute instructions stored in memory in order to carry out one or moreof the image processing methods. In some cases, the methods may beexecuted by a DSP or ISP that invokes various hardware components toaccelerate the image processing. In other cases, the units describedherein may be implemented as, or methods may be performed by, amicroprocessor, one or more application specific integrated circuits(ASICs), one or more field programmable gate arrays (FPGAs), or anycombination thereof.

The steps of a method or algorithm described in connection with theembodiments disclosed herein may be embodied directly in hardware, in asoftware module executed by a processor, or in a combination of the two.A software module may reside in random access memory (RAM), flashmemory, read-only memory (ROM), programmable read-only memory (PROM),erasable programmable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), registers, hard disk, aremovable disk, a compact disc read-only memory (CD-ROM), or any otherform of non-transitory computer readable storage medium known in theart. An exemplary storage medium is coupled to the processor such thatthe processor can read information from, and write information to, thestorage medium. In the alternative, the storage medium may be integralto the processor. The processor and the storage medium may reside in anapplication-specific integrated circuit (ASIC). The ASIC may reside in acomputing device or a user terminal. In the alternative, the processorand the storage medium may reside as discrete components in a computingdevice or user terminal.

The previous description of the disclosed embodiments is provided toenable a person skilled in the art to make or use the disclosedembodiments. Various modifications to these embodiments will be readilyapparent to those skilled in the art, and the principles defined hereinmay be applied to other embodiments without departing from the scope ofthe disclosure. Thus, the present disclosure is not intended to belimited to the embodiments shown herein but is to be accorded the widestscope possible consistent with the principles and novel features asdefined by the following claims.

1. A method comprising: providing a common control signal to multipleimage sensors to be synchronized; receiving a first data line from afirst image sensor of the multiple image sensors; receiving a seconddata line from a second image sensor of the multiple image sensors; andcombining the first data line and the second data line to generate asynchronized data line.
 2. The method of claim 1, wherein thesynchronized data line forms part of a frame, wherein the framecomprises: a first section including first image data from the firstimage sensor; and a second section including second image data from thesecond image sensor.
 3. The method of claim 2, wherein the frame furthercomprises a gap section comprising non-image data disposed between thefirst section and the second section.
 4. The method of claim 2, whereinthe frame has a plurality of rows, wherein each row corresponds to aline index value and stores a line of the first image data having theline index value and stores a corresponding line of the second imagedata having the line index value.
 5. The method of claim 1, wherein thecommon control signal comprises a common clock signal and a common resetsignal.
 6. The method of claim 1, wherein the first data line isassociated with a first pixel size and a first pixel clock, wherein thesecond data line is associated with a second pixel size and a secondpixel clock, and wherein the synchronized data line has a sizesubstantially double that of the first data line or the second data lineand is associated with a pixel clock having a rate that is substantiallydouble a rate of the first pixel clock or the second pixel clock.
 7. Themethod of claim 1, further comprising interleaving a first data streamreceived from the first image sensor and a second data stream receivedfrom the second image sensor on a line by line basis, wherein the firstdata stream includes the first data line, wherein the second data streamincludes the second data line, and wherein combining the first data lineand the second data line is performed during the interleaving of thefirst and second data streams.
 8. The method of claim 1, furthercomprising receiving a third data line from a third image sensor of themultiple image sensors, wherein the third data line is combined with thefirst data line and the second data line to generate the synchronizeddata line.
 9. The method of claim 1, wherein receiving the first dataline is completed before receiving the second data line is completed,and wherein the synchronized data line is generated after receiving thesecond data line is completed.
 10. A device comprising: a first inputconfigured to receive a first data line from a first image sensor ofmultiple image sensors to be synchronized via a common control signal; asecond input configured to receive a second data line from a secondimage sensor of the multiple image sensors; and a combiner coupled tothe first input and to the second input, wherein the combiner isconfigured to combine the first data line and the second data line togenerate a synchronized data line.
 11. The device of claim 10, whereinthe synchronized data line forms part of a frame, wherein the framecomprises: a first section including first image data from the firstimage sensor; and a second section including second image data from thesecond image sensor.
 12. The device of claim 11, wherein the framefurther comprises a gap section comprising non-image data disposedbetween the first section and the second section.
 13. The device ofclaim 11, wherein the frame has a plurality of rows, wherein each rowcorresponds to a line index value and stores a line of the first imagedata having the line index value and stores a corresponding line of thesecond image data having the line index value.
 14. The device of claim10, wherein the common control signal comprises a common clock signaland a common reset signal.
 15. The device of claim 10, wherein the firstdata line is associated with a first pixel size and a first pixel clock,wherein the second data line is associated with a second pixel size anda second pixel clock, and wherein the synchronized data line has a sizesubstantially double that of the first data line or the second data lineand is associated with a pixel clock having a rate that is substantiallydouble a rate of the first pixel clock or the second pixel clock. 16.The device of claim 10, wherein the combiner is configured to interleavea first data stream received from the first image sensor and a seconddata stream received from the second image sensor on a line by linebasis, wherein the first data stream includes the first data line,wherein the second data stream includes the second data line, andwherein the combiner is configured to combine the first data line andthe second data line while interleaving the first and second datastreams.
 17. The device of claim 10, further comprising a third inputcoupled to the combiner and configured to receive a third data line froma third image sensor of the multiple image sensors, wherein the combineris further configured to combine the third data line with the first dataline and the second data line to generate the synchronized data line.18. The device of claim 10, wherein combiner comprises a buffer andwherein the combiner is configured to buffer least a portion of a nextdata line that is received from the first image sensor or from thesecond image sensor before the synchronized data line is generated. 19.The device of claim 10, wherein the first input, the second input, andthe combiner are integrated in a portable electronic device.
 20. Thedevice of claim 10, wherein the first input, the second input, and thecombiner are integrated in a camera module configured to be coupled to aprocessor of a portable electronic device.
 21. A device comprising:means for receiving a first data line from a first image sensor ofmultiple image sensors to be synchronized via a common control signal;means for receiving a second data line from a second image sensor of themultiple image sensors; and means for combining the first data line andthe second data line to generate a synchronized data line.
 22. Thedevice of claim 21, further comprising means for buffering at least aportion of a next data line that is received from the first image sensoror from the second image sensor before the synchronized data line isgenerated.
 23. A device comprising: a first image sensor and a secondimage sensor configured to be synchronized via a common control signal;and a combiner coupled to receive a first data line of the first imagesensor and a second data line from a second image sensor, wherein thecombiner is configured to combine the first data line and the seconddata line to generate a synchronized data line.
 24. The device of claim23, further comprising: an image signal processor coupled process thesynchronized data line; and a display device configured to displayprocessed image data.
 25. A non-transitory computer readable-mediumcomprising instructions, which when executed by a processor cause theprocessor to: provide a common control signal to multiple image sensorsto be synchronized; receive a first data line from a first image sensorof the multiple image sensors; receive a second data line from a secondimage sensor of the multiple image sensors; and combine the first dataline and the second data line to generate a synchronized data line. 26.The non-transitory computer readable-medium of claim 23, wherein thesynchronized data line forms part of a frame, and wherein the framecomprises: a first section including first image data from the firstimage sensor; and a second section including second image data from thesecond image sensor.
 27. The non-transitory computer readable-medium ofclaim 23, wherein the frame further comprises a gap section comprisingnon-image data disposed between the first section and the secondsection.